Goldfedder Building a Data Integration Team

Skills, Requirements, and Solutions for Designing Integrations
1. ed
ISBN: 978-1-4842-5653-4
Verlag: Apress
Format: PDF
Kopierschutz: 1 - PDF Watermark

E-Book, Englisch, 257 Seiten

Find the right people with the right skills. This book clarifies best practices for creating high-functioning data integration teams, enabling you to understand the skills and requirements, documents, and solutions for planning, designing, and monitoring both one-time migration and daily integration systems.
The growth of data is exploding. With multiple sources of information constantly arriving across enterprise systems, combining these systems into a single, cohesive, and documentable unit has become more important than ever. But the approach toward integration is much different than in other software disciplines, requiring the ability to code, collaborate, and disentangle complex business rules into a scalable model. Data migrations and integrations can be complicated. In many cases, project teams save the actual migration for the last weekend of the project, and any issues can lead to missed deadlines or, at worst, corrupted data that needs to be reconciled post-deployment. This book details how to plan strategically to avoid these last-minute risks as well as how to build the right solutions for future integration projects.
What You Will Learn Understand the 'language' of integrations and how they relate in terms of priority and ownershipCreate valuable documents that lead your team from discovery to deploymentResearch the most important integration tools in the market todayMonitor your error logs and see how the output increases the cycle of continuous improvementMarket across the enterprise to provide valuable integration solutions
Who This Book Is For The executive and integration team leaders who are building the corresponding practice. It is also for integration architects, developers, and business analysts who need additional familiarity with ETL tools, integration processes, and associated project deliverables.

Jarrett Goldfedder is the founder of InfoThoughts Data, LLC, a company that specializes in data management, migration, and automation. He has significant experience in both cloud-based and on-premise technologies and holds various certificates in Salesforce Administration, Dell Boomi Architecture, and Informatica Cloud Data. He also served as a technical reviewer of the Apress book by David Masri titled Developing Data Migrations and Integrations with Salesforce: Patterns and Best Practices.

Autoren/Hrsg.

Goldfedder, Jarrett

Inhaltsverzeichnis

1;Table of Contents;5
2;About the Author;12
3;About the Technical Reviewer;13
4;Acknowledgments;14
5;Introduction;16
6;Chapter 1: Integration Background;28
6.1;The Current Landscape: Two Distinct Models;29
6.2;Waterfall Vs. Agile Models;30
6.2.1;The Waterfall Model;30
6.2.2;The Agile Model;32
6.3;Integration Process Vs. Integration Product Ownership;34
6.4;Integration Approaches;35
6.4.1;One-Time Migration;35
6.4.2;Nightly Integration;36
6.4.2.1;Scheduled Service;37
6.4.2.2;Web Service;37
6.4.3;Hybrid Approach;39
7;Chapter 2: Key Terms;40
7.1;The Simplicity of Migration;40
7.2;Source System;42
7.2.1;Connection Type;43
7.2.2;Source Endpoints;43
7.2.3;Metadata Model;44
7.2.4;Data Profiling;45
7.2.4.1;Structure Discovery;45
7.2.4.2;Content Discovery;45
7.2.4.3;Relationship Discovery;45
7.3;ETL Tool;47
7.3.1;Source-to-Target Mapping;47
7.3.2;Design Code;47
7.3.3;Business Rules;48
7.3.4;Exception Handling;48
7.3.5;Version Control;49
7.3.6;Scheduling;49
7.3.7;Logging;50
7.4;Target;50
7.4.1;Target Endpoint;51
7.4.2;Data Matching;51
8;Chapter 3: Team Qualifications;53
8.1;Team Personality;54
8.2;Types of Teams;55
8.2.1;Basic Team Flow;55
8.2.2;Functional;56
8.2.3;Cross-Functional;57
8.2.4;Virtual Team;59
8.2.5;Self-Directed Team;60
8.3;Team Sizes;61
8.4;Waterfall Vs. Agile Team Expectations;62
8.4.1;Waterfall;62
8.4.2;Agile;62
8.4.3;Best Approaches;62
8.4.4;The T-Shaped Person;63
8.5;Integration Team Roles;64
8.5.1;Project Sponsor;64
8.5.2;Stakeholders;64
8.5.3;Subject Matter Expert (SME);64
8.5.4;Product Owner;65
8.5.5;Project Manager (PM);65
8.5.6;Technical Lead;65
8.5.7;Data Integration Architect;65
8.5.8;Software Developers;66
8.5.9;Software Testers;66
8.5.10;User Acceptance Testers;66
8.5.11;Business Analyst;66
8.5.12;Release Manager;67
8.6;Recruiting Your Team;67
8.6.1;Determining the Roles;68
8.6.2;The Job Description;68
8.7;Finding Candidates;70
8.7.1;Internal Candidates;70
8.7.2;External Candidates;71
8.7.2.1;Referrals;71
8.7.2.2;Outsourcing;71
8.7.2.3;Posting to Job Boards;72
8.7.2.4;Hiring a Recruiter;72
8.7.3;Networking;72
8.7.4;Conducting Interviews;73
8.8;Summary;73
9;Chapter 4: Finding Your Purpose: Project Deliverables;75
9.1;Introduction;75
9.2;Project Management Phases;76
9.2.1;Initiation;77
9.2.2;Definition;77
9.2.3;Design Phase;77
9.2.4;Development;78
9.2.5;Implementation;78
9.2.6;Maintenance;80
9.3;Integration Milestones;80
9.3.1;Initiation;82
9.3.2;Definition;83
9.3.2.1;Data Migration Plan;84
9.3.2.1.1;The Discovery Process: Tips and Tricks;86
9.3.2.1.1.1;Learn the Architecture;86
9.3.2.1.1.2;Keep Your Session Small;86
9.3.2.1.1.3;Get System Access Early;87
9.3.2.2;Design;87
9.3.2.2.1;Data Profile Analysis;87
9.3.2.2.2;Issues Log;89
9.3.2.2.3;Source-to-Target Mapping;89
9.3.2.2.3.1;Purpose;89
9.3.2.2.3.2;Design;90
9.3.2.2.3.3;Usage;92
9.3.2.3;Data Migration Process Flow Design;92
9.3.2.3.1;Order of Operations;92
9.3.2.3.2;Mapping Logic;92
9.3.2.3.3;Deployment Steps;93
9.3.2.3.4;Log File Delivery;94
9.3.2.3.5;Migration Results;94
9.3.2.3.6;Final Signoff;94
9.3.2.4;ETL Scripts;95
9.3.2.5;Migration Results Report;95
9.3.2.6;Go-Live/Deployment Plan;96
9.4;Post-deployment: Maintenance and Updates;97
9.5;Summary;98
10;Chapter 5: Choosing an ETL Tool;100
10.1;Introduction;100
10.2;ETL Vs. ELT;100
10.3;ETL As Middleware;102
10.4;The Components of ETL: Choosing the Right Tool;103
10.4.1;Extraction;103
10.4.2;Establishing Connectors;104
10.4.3;Storing Temporary Data;104
10.4.4;Validating Inbound Data;105
10.5;Transformation;105
10.5.1;Modifying Data;106
10.5.2;Analyzing Data Integrity;106
10.5.3;Validating Result Data;109
10.6;Loading;110
10.6.1;Establishing Connectors;111
10.6.2;Determining Load Types;113
10.6.3;Rolling Back Errors;113
10.7;Tools on the Market;114
10.7.1;Jitterbit;115
10.7.2;Talend;116
10.7.3;Dell Boomi;117
10.7.4;Pentaho Data Integration (PDI);119
10.7.5;Microsoft SQL Server Integration Services (SSIS);121
10.7.6;MuleSoft;122
10.7.7;Informatica;123
10.7.8;Scripting Languages;125
10.8;Summary;126
11;Chapter 6: A Sample ETL Project;127
11.1;Introduction;127
11.2;Project Management Phases Revisited;128
11.2.1;Initiation;128
11.2.2;Definition;128
11.2.3;Design;133
11.2.3.1;Data Profiling;133
11.2.3.2;Source-to-Target Mapping;134
11.2.3.3;Data Migration Process Flow Design;140
11.2.4;Development and Implementation;141
11.2.4.1;Getting Ready;142
11.2.4.1.1;Clone the GitHub Directory;142
11.2.4.1.2;Download and Install PDI (Community Edition);142
11.2.4.2;Opening the “create_target_table_buyer” Script;144
11.2.4.3;Opening the “Prototype” Script;146
11.3;Summary;149
12;Chapter 7: Platform Automation;151
12.1;Introduction;151
12.2;All About Environments;152
12.2.1;Why It Matters;152
12.2.2;Types of Environments;154
12.2.2.1;Development;154
12.2.2.2;Staging;155
12.2.2.3;Production;156
12.2.3;Environments for One-Time Migration;156
12.2.4;Environments for Repeatable Integrations;158
12.2.5;DevOps Practices: A Sample Deployment;159
12.2.5.1;Our Architecture;160
12.2.5.2;Preparing for Automation;162
12.2.5.3;Proposed Migration and Integration Approach;165
12.2.5.4;Demonstration;167
12.2.5.5;The Output Step;172
12.3;Rollback;174
12.4;Testing the Automation;175
12.4.1;Testing;176
12.5;Job Scheduling;179
12.6;Summary;185
13;Chapter 8: Monitoring Results;187
13.1;Introduction;187
13.2;Day One: Post-deployment;188
13.2.1;Level of Support;188
13.2.2;Common Integration Complaints and Solutions;190
13.2.3;The Daily Data Log;193
13.2.4;Updating Production;193
13.3;Week One: Post-deployment;195
13.3.1;Planning Your Next Steps;196
13.3.2;Continuous Improvement;196
13.4;Month One: Post-deployment;204
13.5;The Data Management Maturity (DMM) Model;204
13.6;Year One: Post-deployment;208
13.7;MDM;209
13.8;Summary;210
14;Chapter 9: Marketing Your Team;211
14.1;Introduction;211
14.2;Addressing Leadership;212
14.3;Educating Others;214
14.3.1;Internal Education;215
14.3.1.1;Formal Seminars;215
14.3.1.2;Workshops;216
14.3.1.3;Online Training;217
14.3.1.4;Lunch and Learns;218
14.3.2;External Education;220
14.3.2.1;Meetups;221
14.3.2.2;Networking Event;222
14.3.2.3;Conferences;223
14.4;The Data Integration Roadmap;224
14.5;Data Governance and the Data Governance Office;228
14.6;Summary;230
15;Appendix A:Data Migration Strategy Plan;231
15.1;Overview;231
15.1.1;Introduction;231
15.1.2;Purpose;231
15.1.3;References;232
15.2;Assumptions, Constraints, Risks, and Gaps;232
15.2.1;Assumptions;232
15.2.2;Constraints;232
15.2.3;Risks;233
15.2.4;Gaps;233
15.3;Architecture;233
15.3.1;Current State;233
15.3.1.1;Current State Data Sources;234
15.3.1.2;Current State Record Counts;234
15.3.1.3;Current State Data Model;234
15.3.1.4;Current State Integration Points;234
15.3.2;Future State;234
15.3.2.1;Future State Data Sources;234
15.3.2.2;Future State Data Model;234
15.3.2.3;Future State Integration Points;235
15.4;Development Tools;235
15.4.1;Environment Usage;235
15.4.2;Data Migration Approach;235
15.4.2.1;Scope;235
15.4.2.2;Approach;235
15.4.2.3;Team Roles and Responsibilities;236
15.4.3;Migration Process;236
15.4.3.1;Pre-migration Activities;236
15.4.3.2;Migration Activities;236
15.4.3.3;Post-Migration Activities;236
15.5;Contingency Plan;236
15.6;Testing/Validation;236
15.6.1;Testing Methods;237
15.6.2;Migration;237
15.6.3;Integration;237
16;Appendix B:Data Profiling Template;238
17;Appendix C:Issues Log Template;240
18;Appendix D:Source-to-Target Mapping Template;241
19;Appendix E:Data Migration Process Flow Design Template;244
19.1;Order of Operation;244
19.2;Mapping Logic;245
19.3;Deployment Steps;246
19.3.1;Description of Prerequisites;248
19.3.2;Contact;248
19.4;Account;248
19.4.1;Project Signoff/Document Delivery;248
19.5;Log File Delivery;249
19.5.1;Individual Delivery;249
19.5.2;Portal Storage;249
19.6;Migration Results;249
19.6.1;Contact;249
19.6.1.1;Inserts;249
19.6.1.2;Updates;250
19.6.2;Account;250
19.7;Final Signoff;250
20;Appendix F:Migration Results Report Template;251
21;Index;253