Hi, I'm Mayank Agarwal

Core Architecture & Data Flow:

• Built end-to-end Spring Boot backend consuming high-volume restriction events from multiple upstream Kafka topics
• Designed real-time decision engine to evaluate incoming account restriction signals and determine add/remove actions for associated debit cards
• Integrated with multiple cross-team APIs via secure API networks for deep account metadata and validation

High-Scale Data Processing:

• Designed new MS SQL schema optimized for daily ingestion of 2-3 million debit card records
• Implemented batch + streaming hybrid processing with multi-stage validation pipelines
• Achieved greater than 99.9% accuracy in restriction application while handling millions of records daily

Cloud-Native Deployment:

• Deployed entire system on Kubernetes using GitOps and Helm charts
• First full-fledged banking backend application at MS to run fully on Kubernetes
• Improved deployment reliability by approximately 60% and reduced rollback time from hours to minutes

Impact:

• Reduced manual intervention in card restriction workflows by 70%+
• Improved response time for risk enforcement from hours to near real-time

Daily Aggregation & Calculation Engine:

• Built high-precision computation engine that aggregated all irregularities and claims per account for a given day
• Merged multiple events into single net settlement value with complex mathematical and financial rules
• Generated audit-ready memo files for downstream financial systems and compliance teams

Event-Driven Settlement Pipeline:

• Produced Kafka events to downstream settlement systems with validated calculations
• Developed dedicated Booking Engine to credit amounts to creditor accounts with manual Ops approval
• Integrated with Ops Kafka topics to maintain clear approval and confirmation trail

Vendor & MQ Integration:

• Built custom MQ-based integration from scratch to notify Debit Card Vendor once credits were approved
• Ensured reliable confirmation with exactly-once delivery semantics between internal systems and external vendors

Impact:

• Reduced claim settlement processing time by approximately 45%
• Eliminated reconciliation mismatches caused by fragmented calculations
• Improved auditability and compliance confidence across financial operations

Large-Scale File Processing:

• Ingested massive vendor files containing data for millions of debit cards
• Engineered memory-efficient, chunked processing pipeline optimized for time and space complexity
• Enabled parallel validation streams while avoiding JVM memory pressure

Data Validation & Integrity:

• Performed multi-field verification on every record: card attributes, account mappings, vendor consistency checks
• Stored only fully verified and approved records in newly designed MS SQL tables

Tokenization & Privacy Enhancement:

• Generated tokenized card numbers and built downstream pipelines to replace raw card numbers with tokens
• Distributed tokenized datasets (millions of cards) to dependent teams across internal systems

Impact:

• Improved customer data privacy by greater than 90% reduction in raw PAN exposure
• Enabled seamless vendor migration with zero downtime
• Established reusable framework for future migrations and compliance initiatives

Event-Driven Data Pipeline:

• Built event-driven balance validation pipeline ingesting data from multiple upstream systems via Kafka streams and large batch files
• Ensured fault-tolerant and scalable processing with distributed design patterns
• Designed Kafka-based search and replay mechanisms to recompute balances on demand, enabling investigation of discrepancies without data loss

Multi-Layer Reconciliation Engine:

• Implemented multi-layer reconciliation engine to segregate market/trade-in-progress activity (open positions, unsettled instructions) from actual monetary movements
• Applied domain-specific formulas per transaction category with strict validation rules before generating final aggregate account state
• Persisted validated balances, transaction trails, and reconciliation metadata in MS SQL Server, optimized for both low-latency reads and historical audits

Full-Stack UI & Security:

• Built React + TypeScript UI from scratch, displaying real-time account state, historical transaction timelines, recalculation outcomes, and audit traces
• Integrated Kerberos authentication and authorization, enforcing full access only for users with active system entitlements
• Implemented restricted views for standard users (self-account data only) with role-based access control

End-to-End Ownership & Impact:

• Owned complete solution including architecture, backend services, frontend UI, security integration, and production readiness
• Significantly improved balance accuracy and traceability across high-volume financial accounts
• Reduced manual reconciliation effort and investigation time by approximately 35%
• Enabled near real-time discrepancy detection in regulated enterprise environment

High-Performance Data Ingestion:

• Engineered high-performance data ingestion engine capable of processing 2-3 GB Excel files, transforming raw financial records into structured datasets without memory bottlenecks
• Implemented memory-efficient processing to handle large-scale financial data with optimal performance
• Focused heavily on performance optimization while maintaining responsive UI and query execution

Advanced Analytics & Search:

• Designed and populated large analytical tables with advanced filters, multi-column sorting, and full-text and conditional searches
• Built detailed transaction history views for users holding multiple accounts, enabling granular analysis across time ranges and account types
• Developed interactive dashboards, charts, and reports to analyze spending behavior, investment patterns, and account-wise inflows and outflows

Event-Driven Recalculation & Automation:

• Implemented Kafka-backed recalculation workflows to validate balances and regenerate account states under changing inputs
• Automated balance recomputation using Unix scripts, orchestrating validation, recalculation, and data refresh pipelines
• Built end-to-end automation for data validation and reconciliation processes

Impact & Production Adoption:

• Demonstrated scalability, flexibility, and enterprise readiness through comprehensive POC implementation
• POC design directly influenced adoption in full-scale production system built during Technology Analyst role
• Established technical foundation for enterprise-grade balance validation platform
• Gained hands-on experience with big financial datasets, distributed messaging, and reconciliation logic

Redis-Backed In-Memory Architecture:

• Built Redis-backed in-memory trade ledger serving as primary system of record during active trading hours
• Enabled millisecond-level reads/writes for high-frequency trade operations with ultra-low latency
• Achieved high throughput under concurrent access while maintaining strong correctness guarantees expected in financial trading systems

Advanced Concurrency Control:

• Designed strict concurrency control model to prevent duplicate trade bookings and conflicting bookings by multiple users targeting same instrument/order
• Implemented distributed locking mechanisms in Redis using atomic operations (SET NX PX), lock expiration (TTL), and ownership verification
• Introduced fine-grained lock scoping (user-level, instrument-level, order-level locks) to maximize parallelism while guaranteeing trade correctness
• Engineered deadlock-safe execution paths with deterministic lock acquisition order and automatic lock recovery on failures

Trade Lifecycle Management:

• Start of Day (SOD): Loaded latest confirmed trade state from database into Redis with validation for consistency
• Intraday Trading Window: All trade booking, updates, and validations occurred in Redis for ultra-low latency with well-defined state transitions
• End of Day (EOD): Executed controlled, idempotent flush of only confirmed trades from Redis to database with exactly-once persistence semantics
• Used atomic Lua scripts in Redis to bundle multi-step operations (lock, validate, write, unlock) into single execution unit, eliminating race conditions

Impact & Technical Excellence:

• Demonstrated deep understanding of distributed systems fundamentals: consistency vs availability, locking strategies, and failure recovery
• Built production-inspired trading architecture capable of handling real-world concurrency and settlement workflows
• Established patterns reusable for order management systems, high-frequency trading platforms, and financial booking engines

End-to-End UI Development:

• Translated Figma/design mockups into fully functional, responsive Web UI using React.js, Bootstrap, and custom CSS
• Established component architecture for scalability, separating layout, feature, and shared UI components
• Ensured cross-browser compatibility, mobile responsiveness, and accessibility-friendly layouts

Assertifications Module (Certification Flow):

• Designed and implemented certification module enabling users to apply for skill-based tests, track approval status, and view certification eligibility
• Integrated backend APIs (Node.js + Express) to fetch dynamic test data and approval states
• Implemented state-driven UI rendering handling multiple test lifecycle stages (applied, approved, booked, completed)

Checkout & Payment Flow:

• Built secure multi-step checkout flow for test booking: user verification, test selection, payment initiation and confirmation
• Designed fail-safe UI states for payment success, failure, retries, and cancellation
• Ensured clean separation of UI logic and payment orchestration, making flow extensible for future gateways

Intelligent Search & Advanced Filters:

• Implemented smart search engine UI with keyword-based search and multi-filter support (skills, difficulty level, certification type, prerequisites)
• Optimized frontend filtering using memoization and controlled re-renders for performance
• Designed filter logic to align with user skill profiles, improving discoverability of relevant certifications

Test Booking & Proctoring Validation:

• Built Test Booking pages with strict pre-test validation: camera access verification, microphone and voice detection checks
• Integrated real-time media validation to detect camera availability and identify background blur or virtual/fake backgrounds
• Designed UI flows to block test initiation unless all authenticity checks passed, ensuring exam integrity

Performance Optimization & Backend Integration:

• Refactored components into highly reusable modular units, reducing UI duplication
• Optimized state management using controlled state lifting, efficient prop drilling minimization, and memoization techniques
• Worked closely with backend services built on Spring Boot, MongoDB, and Mongoose ODM
• Consumed REST APIs for test discovery, user eligibility checks, booking and payment data

Impact & Outcomes:

• Delivered first working Web UI of Assert platform from scratch
• Enabled end-to-end certification journeys from search to booking to test validation
• Improved platform usability, scalability, and readiness for real-world proctored assessments
• Helped convert early-stage startup designs into production-grade product

On-Device AI Execution Architecture:

• Studied and evaluated edge AI constraints including low-latency inference requirements, memory and compute limitations on mobile/embedded hardware
• Analyzed benefits of local inference vs cloud-based inference: reduced network dependency, enhanced user privacy, and offline AI capability
• Explored power efficiency and real-time execution guarantees for on-device deep learning

Deep Learning Foundations & Optimization:

• Built strong theoretical grounding in perceptrons, layered neural architectures, activation functions, loss functions, and regularization techniques
• Implemented and analyzed optimization algorithms: SGD with bias correction, RMSProp, and Adam optimizer
• Explored CNN internals including convolution operations, padding strategies, and max/average pooling mechanisms

Model Research & Architecture Evaluation:

• Studied modern efficient architectures designed for performance-to-compute trade-offs: RegNet-Y for scalable vision workloads and Wav2Vec 2.0 for speech representation learning
• Evaluated architectural suitability for edge deployment scenarios, balancing accuracy, model size, and inference latency
• Analyzed hardware-software co-design principles for edge AI optimization

Model Conversion & Deployment Pipeline:

• Hands-on experience with TensorFlow, PyTorch for model experimentation, and TensorFlow Lite for mobile inference
• Worked on model conversion to TFLite, evaluating quantization and optimization strategies
• Understood compatibility constraints between TensorFlow to TFLite and TensorFlow/PyTorch to ONNX conversions
• Explored custom operator creation in TensorFlow, parser additions based on execution worklets, and validation of converted models

ARM & Embedded AI Considerations:

• Gained introductory exposure to ARM NN SDK for optimized execution on ARM Cortex processors
• Studied execution flow on ARM Cortex-based devices and how ARM NN bridges ML frameworks with hardware acceleration
• Understood importance of operator support, memory planning, and runtime scheduling for edge AI

Impact & Learning:

• Followed structured weekly milestones with technical reviews and research discussions
• Learned to break down ambiguous research problems and translate papers into executable insights
• Gained comprehensive understanding of AI model lifecycle from training to edge deployment

Kernel-Based CI Testing (RKHS):

• Designed and implemented kernel-based CI tests grounded in Reproducing Kernel Hilbert Space (RKHS) theory to detect non-linear and non-Gaussian dependencies
• Used Gaussian/RBF kernels to map variables into high-dimensional feature spaces where conditional independence can be tested via covariance operators
• Addressed limitations of classical CI tests (e.g., partial correlation) which fail under non-linear relationships

First Python Implementation of RCoT & RIT:

• Delivered first Python implementation of RCoT (Randomized Conditional Correlation Test) and RIT (Randomized Independence Test)
• Implemented Random Fourier Features (RFF) to approximate kernel mappings, reducing complexity from O(n³) to O(n·d)
• Enabled scalable CI testing, making causal discovery feasible on larger datasets

Advanced Distribution Approximation:

• Implemented and compared statistical approximation methods to estimate p-values: LPB (Lindsay-Pilla-Basak) and HBE (Hall-Buckley-Eagleson) approximation
• Solved numerical issues related to eigenvalue instability, skewed test statistic distributions, and high-dimensional covariance matrices
• Improved accuracy and reliability of hypothesis testing, especially in low-sample and high-dimension regimes

Causal Graph Generation Pipeline:

• Integrated CI tests into constraint-based causal discovery workflows (PC-style algorithms)
• Enabled edge pruning based on conditional independence and discovery of direct vs indirect causal relationships
• Constructed causal graphs (DAGs) from observational data, laying groundwork for automated causal structure learning

Performance Optimization & Engineering:

• Optimized linear algebra operations using efficient matrix factorizations and vectorized NumPy workflows
• Achieved significant runtime improvements without sacrificing statistical validity
• Designed modular, reusable Python components suitable for integration into larger analytics frameworks
• Validated correctness against theoretical expectations and benchmark datasets

Research to Engineering Translation:

• Translated academic research papers into clean, maintainable production code
• Documented mathematical foundations, algorithmic assumptions, and implementation trade-offs
• Published technical explanations and implementation insights via internal/external blog

Impact & Outcomes:

• Enabled scalable, non-linear causal discovery for real-world datasets
• Reduced computational bottlenecks in CI testing, a major limitation in causal graph learning
• Strengthened HPCC's Causality Toolkit with practical, research-backed algorithms ready for platform integration

Content Outreach & Marketing:

• Executed targeted outreach campaigns to promote MyCaptain's educational programs across multiple cohorts
• Engaged with prospective students through social media, email campaigns, and community channels to raise awareness about mentorship opportunities
• Supported marketing initiatives to increase program enrollment and student participation

Student Engagement & Support:

• Facilitated student onboarding and orientation processes for new cohorts joining the platform
• Addressed student queries regarding program structure, mentor matching, and learning pathways
• Maintained consistent communication with students to ensure positive learning experiences and program completion

Program Coordination:

• Collaborated with operations team to coordinate learning and development sessions across multiple cohorts
• Assisted in tracking student progress and achievement of learning targets
• Supported mentor-mentee matching processes to align students with appropriate field experts

Impact & Learning:

• Gained hands-on experience in EdTech marketing, student engagement, and community building
• Developed skills in content outreach, digital marketing, and customer relationship management
• Contributed to MyCaptain's mission of democratizing mentorship and career guidance for students
• Learned the importance of personalized outreach and student-centric communication in educational platforms