Engage Group

EN

  • CONTACT
  • SERVICES
    • STRATEGY
    • ADVISORY
    • STEERING
    • SPECIALISTS
    • TESTING
    • DEVOPS
  • NEWS
  • ARTICLES
  • ABOUT US

Export to Data Lake, but what then?

June 2, 2022 by Engage Group

Microsoft is on a mission to provide the most comprehensive “Intelligent Cloud” on the planet. Significant investments have been made related to the Business Applications domain with the tighter integration towards various Azure Data Platform services such as Data Lake and Synapse Analytics as main themes. But what happens once the data lands in the lake and what patterns are commonly used by our customers and why? This article aims to dissect these questions based on our experience and reference patterns provided by Microsoft.    

For an introduction to the “Export to Data Lake” capability for Dynamics 365 F&SCM, Microsoft provides excellent documentation through Docs articles and Tech Talk recordings.

Essentially, once the data is exported from Dynamics 365 and lands in the Lake, it will be stored as .csv files according to the Common Data Model syntax. At this stage, the data provides very limited value due to lack of headers, relationships etc. Instead, all metadata is stored in related .json files. See an example below showcasing Customer table data exported as a .csv file to an Azure Data Lake container.

No alt text provided for this image

Consequently, the question at this stage will be how to create insights from this pile of data?

BI- and Analytics Architecture Patterns

Microsoft outlines three common BI- and Analytics architecture patters in a recent “Scenarios and Architecture Patters” Tech Talk with the intention to provide examples on how to create insights from data:

# 1 – Logical Data Warehouse

# 2 – (Cloud) Data Warehouse

# 3 – Data Lakehouse

While, according to our experience, the Data Warehouse pattern (#2) has been the main approach for most Business Applications clients throughout the last decade, alternative 1 and 3 have received a surge in interest during the last years due to mega trends such as Cloud adoption, Big Data and Artificial Intelligence.

From a tooling perspective, Microsoft caters to the needs of all these three patterns through the comprehensive stack of Azure Data Platform services. However, due to the market interest of the Data Lakehouse patterns we aim to take a closer look at the rationale for adopting this pattern in a Dynamics 365 context.

Data Lakehouse

First of all, let’s clarify what a Data Lakehouse architecture actually is. In the article: “Why a Delta Lakehouse?”, Robert Thompson and Geoff Freeman describes it as:

“A Data Lakehouse is simply a data warehousing principle applied over data in a Data Lake. Data is exposed through delta tables, an open-source protocol for interacting with the data in a lake in an ACID- compliant way”.

This means, instead of loading the data into a central repository such as an SQL database, the data stays in the lake using a flat file structure. Another key difference between the Data Lake and the Data Warehouse is that data lakes store the data without arranging it into any logical relationships which is a key enabler for advanced analytics and AI. See Microsoft illustration below highlighting main components and corresponding Azure Data Platform services:

No alt text provided for this image

Source: Microsoft

Key concepts of this pattern are described below:

Ingest – Extract data from on-prem and cloud data sources and load into the Data Lake

Enrich, Transform – Key component of the Data Lakehouse architecture. Here, data is divided into different zones depending on to what extent the data has been shaped to cater for end-user needs. Typically, three “zones” are used:

  • Bronze – Landing zone for all data ingested into the Data Lake. Either stored as-is for batch patterns or as aggregated datasets for streaming workloads
  • Silver – Filtered and enriched data for exploration purposes according to business needs. Stores the latest state of the data.
  • Gold – Curated, well-structured data compliant with the organizations BI-tools and AI-algorithms. Moreover, this zone could also be used for feeding a traditional Data Warehouse during a transition period.

Visualize – Exposes the data in a way that make sense to the end-users through reports and dashboards

Features and Tools – Related to Microsoft and their investments to provide a seamless analytics experience for Dynamics 365 customers. Examples being the native service for automatically exporting Dynamics 365 data to the Data Lake in near-real time. Another example is the CDMUtil-tool for materializing views based on Dynamics 365 data using Synapse Analytics according and the Common Data Model schema.

The data lakehouse pattern is a relatively new concept in the data analytics scene. It was first introduced in 2017 in relation to the Snowflake platform. In 2020 the term became widely known when adopted by Databricks for its Delta Lake platform.

Benefits and Limitations with the Data Lakehouse patters

As always there are tradeoffs to be made when choosing between different BI- and Analytics patterns. Below is a list of examples based on our experience.

Benefits

  • Cost efficient cloud storage solution for Big Data scenarios
  • Flexible solution since structured, semi-structured and un-structured data could be managed
  • Strong support for Data science and Machine Learning experimentation

Limitations

  • Risk of creating a “data swamp” rather than a Data Lake if proper governance is not established
  • Good fit for cloud environments but not for on-prem due to main concept of separating compute and storage
  • Considered as a relatively new (immature) technology and less adopted by organizations which could require additional training and technology assessment activities

Success factors during implementation

Whether you want to modernize legacy systems to support advanced analytics scenarios or build a new BI- and Analytics solution from scratch, there are key considerations to be evaluated.

First, ensure the right mix of competence. Dynamics 365 specialists for business process- and data model understanding including how to configure the native export of data to the Data Lake. BI- and Analytics specialist to realize your target architecture. If using partners, this typically involves onboarding separate niche players for Dynamics 365- and Analytics competence.    

Secondly, familiarize yourself with the core concepts and available standard capability. Preferably, by executing a Proof-of-Concept in a Sandbox environment where a simple hypothesis is validated. For example, export Dynamics 365 data to the lake and then apply a compute service such as Apache Spark by creating a Notebook to curate the data and then finally visualize insights in a report using Power BI.

Finally, ask yourself why you need a data storage solution? Is it for regulatory reporting, business intelligence, real-time analytics and/or data science? Ensure that your analytics needs are well defined to ensure you are targeting the correct personas across your organization. Considering Dynamics 365 F&SCM, the workload ships with a vast number of reporting and BI-capabilities to cater for various needs. Exporting the data to a data lake for further processing is only one available option. Understanding which capabilities that is available out-of-the-box will ensure you maximize the value of your existing Dynamics 365 investments.   

Would you like to know more, contact us at: contact@engagenow.com

Filed Under: Articles

Agile for ERP

March 21, 2022 by Engage Group

Finally, the agile mindset and approach has reached the more conservative ERP world. At Engage, we have been involved in several such projects where it has been proven that it not only works but also has great advantages over the traditional waterfall method of ERP implementation.

We do see strong advantages in adapting to agile principles, ways-of-working and tools:

• Increase productivity
• Improve collaboration (both IT and Business)
• Increase transparency to priorities and backlog
• Decrease risks

“Waterfall vs Agile” is not always helpful as a “one or the other“ distinction in the new cloud world but we do find some key items to be essential for success:

• Choose the best team for the task – easier said than done if you are in the hands of an outsourcing partner
• If possible, have a Minimum Viable Product approach
• Make sure to have a process for handling changed and new priorities
• Early and repeated validation of the solution to make sure that processes and scenarios meet the business needs and requirements
• Make sure to get ambassadors among the business stakeholders to gain confidence, experience and increase engagement

Organizations that are the most successful in embracing agile, understand that agile methodologies are not something that organizations do, but rather something that the organization becomes. It takes a long time to become agile, but the payoff is proven even in the ERP-world for application lifecycle management.

Filed Under: Articles

Senior Specialist SCM and PM

March 7, 2022 by Engage Group

We are proud to announce that Drew Fremlev Fisher joins Engage Group DK (Copenhagen office) from the 1st of March 2022.

Drew has a strong background within ERP & Retail, having worked for internationally recognised brands like Georg Jensen and Pandora for more than a decade, where his focus has been on building efficient supply chain processes. In addition, he is no stranger to Project Management where he has earned his merits leading through multiple ERP implementation projects.

He was born and raised in Australia but has lived in Denmark for more than 7 years. He loves learning new things and says that: “My drive comes from creating positive change, seeing my actions and work, both alone and in collaboration with others, producing results.”

Drew has already been booked to his first assignment by a global retail customer to help improve their overall SCM and Inventory Management processes.

We are very happy and proud to have you onboard. Welcome, Drew!

Filed Under: News

Global Roll Out Assignment to Engage Gothenburg

February 23, 2022 by Engage Group

Engage Group selected as strategic partner to build and globally roll out a template in the manufacturing industry.

Engage has an experienced and efficient team delivering with the customer in a large number of processes including, Finance, Manufacturing, Service and Supply Chain.

Although the implementation of the entire Dynamics 365 platform is in focus, the team also handles change management and the introduction of new ways of working.

“Being part of a truly global team working for a common goal in a positive atmosphere is extremely satisfying and our team is really glad to be part of this journey. The strong management commitment, the capabilities of the D365 platform (CRM/ERP) together with a senior team make the high pace for the roll out possible” says Hans Gillesén, Partner at Engage Group.

Filed Under: News

Helene Sahlsten Appointed Head of Delivery

February 22, 2022 by Engage Group

Helene will help Engage Group´s customers to achieve true business value through the successful delivery of projects and managed services based on the Microsoft Dynamics 365 product portfolio and proven agile implementation methods. Putting quality and progress in the first palace, Helene’s long experience of mission critical ERP solutions will ensure our customers a proactive and secure delivery.

“I am thrilled to join Engage as Head of Delivery and to be part of a company which understands that what we are really working with are relationships. All trust, transparent dialogue and willingness of commitment comes from a solid and genuine relationship. That is why Engage Group always has a strong customer presence and customer success as key focus. Together with our brilliant teams, stakeholders and customers I will have focus on taking the next step with our delivery and together we will create success stories”, says Helene Sahlsten.

“Helene is a perfect fit to Engage as we always put the customer in focus and want be a proactive advisory partner drawing from our collective experience. Helene’s leader and service capabilities will further strengthen our PMO and service deliveries”, says Stefan Jegbert, Partner and Head of Advisory Services at Engage Group

Filed Under: News

New Lead Solution Architecture Assignment

February 17, 2022 by Engage Group

We are glad to announce that we have been contracted to develop and govern the global solution blueprint at an international process industry client. This is a strategic assignment for our newly established presence in Skåne.

The strategic choice to establish Engage Group in the south of Sweden prove to be an imminent success. The market for Dynamics 365 consultants is overheated as it is but the unique concept that Engage Group brings to the table is equally attractive for both consultants and clients.

Filed Under: News

Specialist – Finance

February 17, 2022 by Engage Group

This week, Sara Rundberg joins the finance team in Gothenburg.

“I have during many years supported and improved Financial processes, where the latest years have been focused on implementing ERP solutions and Dynamics 365. It now feels like an exciting and natural step to deepen and broaden my competence at Engage group among the best Dynamics consultants on the market”. says Sara

“Having Sara as part of the team will secure we have best possible Financial Business Processes understanding with an extra deep knowledge of AP Automation and Bank Payments” says Hans Gillesén Partner at Engage.

Filed Under: News

Store Inventory Counting Assignment

February 14, 2022 by Engage Group

Engage Group has been appointed to perform a store inventory counting advisory assessment for a major Nordic fashion retailer. The objective is to leverage Dynamics 365 Commerce built-in capabilities to ensure a robust and future proven solution.  

“At Engage, we have a long experience working together with clients in the fashion retail industry. This assignment allows us to share our experience to ensure a user-centric solution where business requirements are addressed by standard capabilities and aligned with future Microsoft Roadmap”, says Tobias Lång, CTO at Engage Group. 

“Microsoft is investing heavily in the stock counting experience for Dynamics 365 Commerce with new features recently released such as optimized barcode scanning, in-parallel counting, review possibilities and performance improvements. In addition, a new “Store Commerce”- app has been released to Preview.  We look forward to utilizing these capabilities to ensure our client is maximizing their current Dynamics 365 investment”, says Henrik Christenson, CEO and Partner at Engage Group. 

Filed Under: News

Engage Group announces the opening of new office in Malmö

February 10, 2022 by Engage Group

Engage Group has quickly been established as the leading player for Dynamics in the Nordic region with offices in Stockholm, Gothenburg, Oslo, Copenhagen, Aarhus, Helsingborg and now Malmö. In one of Malmö’s most exciting places, in one of the city’s most vibrant houses, is Engage´s southernmost arena.

“We continue to see great demand for our Microsoft Dynamics services in the Nordic region and are pursuing our successful strategy of being close to the local market”, says Henrik Hasselblad, Managing Partner at Engage Group.

“Engage is truly the most exciting Dynamics company I have come across, with a culture that instead of focusing on reactive consulting, is aiming at being the best advisor in a true partnership with their customers. I am very much looking forward to bringing Engage to Skåne”, says Anders Rylander CEO at Engage Group South.

Welcome to us at Västra Hamnen and the Studio building!

Filed Under: News

Completed Rapid Cloud Transition at major Retailer

February 4, 2022 by Engage Group

Rapid Cloud Transition assignment completed for Global Retailer 

Engage Group has recently completed a Rapid Cloud Transition assignment at a global retailer. The objective has been to prepare the organization for an upgrade to Microsoft Dynamics 365, both in terms of providing input to the business case but also to ensure that the client is leveraging the vast amount of innovation continuously released by Microsoft to create long term business value.  

The Rapid Cloud Transition is a framework developed by Engage Group to address a significant market demand and consist of the following key pillars: 

  • Solution Strategy Assessment – Application Portfolio Management artefacts to ensure Dynamics 365 fits into the organization’s target architecture 
  • Business Value Assessment – Anchor business value by demonstrating how Dynamics 365 could address current challenges 
  • Upgrade Assessment – A review of current code base, related KPI:s and recommended upgrade path 
  • Decision Material – Input to business case such as sourcing model, resource needs and high-level project plan 

“A lift-and-shift to the cloud for critical workloads such as the ERP-system demands a shift in mindset since it has a major impact on the entire organization. By adopting a structured approach such as the “Rapid Cloud Transition”, one takes a holistic approach to the digital transformation journey rather than viewing it as pure IT-project significantly improving the success rate.”  – Tobias Lång, CTO Engage Group 

“We have refined our offering over the past years and now have a market-leading concept for building a strong business case for the Dynamics 365 platform. In the process, we have the privilege of getting to know key employees and provide well anchored decision material to top management. This gives us a unique relationship with the customer that lasts over time where we mix management consulting with a deep understanding of the business applications from Microsoft.” – Henrik Hasselblad, Managing Partner Engage Group

Filed Under: News

  • 1
  • 2
  • Next Page »

Follow us

  • LinkedIn
  • YouTube

Contact us

  • Email

Experts in Dynamics 365

Accelerate your cloud journey with Dynamics 365 expertise and resources

Copyright © 2023 · Agency Pro on Genesis Framework · WordPress · Log in