Articles Archives - Engage Group

Export to Data Lake, but what then?

June 2, 2022 by Engage Group

Microsoft is on a mission to provide the most comprehensive “Intelligent Cloud” on the planet. Significant investments have been made related to the Business Applications domain with the tighter integration towards various Azure Data Platform services such as Data Lake and Synapse Analytics as main themes. But what happens once the data lands in the lake and what patterns are commonly used by our customers and why? This article aims to dissect these questions based on our experience and reference patterns provided by Microsoft.

For an introduction to the “Export to Data Lake” capability for Dynamics 365 F&SCM, Microsoft provides excellent documentation through Docs articles and Tech Talk recordings.

Essentially, once the data is exported from Dynamics 365 and lands in the Lake, it will be stored as .csv files according to the Common Data Model syntax. At this stage, the data provides very limited value due to lack of headers, relationships etc. Instead, all metadata is stored in related .json files. See an example below showcasing Customer table data exported as a .csv file to an Azure Data Lake container.

Consequently, the question at this stage will be how to create insights from this pile of data?

BI- and Analytics Architecture Patterns

Microsoft outlines three common BI- and Analytics architecture patters in a recent “Scenarios and Architecture Patters” Tech Talk with the intention to provide examples on how to create insights from data:

# 1 – Logical Data Warehouse

# 2 – (Cloud) Data Warehouse

# 3 – Data Lakehouse

While, according to our experience, the Data Warehouse pattern (#2) has been the main approach for most Business Applications clients throughout the last decade, alternative 1 and 3 have received a surge in interest during the last years due to mega trends such as Cloud adoption, Big Data and Artificial Intelligence.

From a tooling perspective, Microsoft caters to the needs of all these three patterns through the comprehensive stack of Azure Data Platform services. However, due to the market interest of the Data Lakehouse patterns we aim to take a closer look at the rationale for adopting this pattern in a Dynamics 365 context.

Data Lakehouse

First of all, let’s clarify what a Data Lakehouse architecture actually is. In the article: “Why a Delta Lakehouse?”, Robert Thompson and Geoff Freeman describes it as:

“A Data Lakehouse is simply a data warehousing principle applied over data in a Data Lake. Data is exposed through delta tables, an open-source protocol for interacting with the data in a lake in an ACID- compliant way”.

This means, instead of loading the data into a central repository such as an SQL database, the data stays in the lake using a flat file structure. Another key difference between the Data Lake and the Data Warehouse is that data lakes store the data without arranging it into any logical relationships which is a key enabler for advanced analytics and AI. See Microsoft illustration below highlighting main components and corresponding Azure Data Platform services:

Source: Microsoft

Key concepts of this pattern are described below:

Ingest – Extract data from on-prem and cloud data sources and load into the Data Lake

Enrich, Transform – Key component of the Data Lakehouse architecture. Here, data is divided into different zones depending on to what extent the data has been shaped to cater for end-user needs. Typically, three “zones” are used:

Bronze – Landing zone for all data ingested into the Data Lake. Either stored as-is for batch patterns or as aggregated datasets for streaming workloads
Silver – Filtered and enriched data for exploration purposes according to business needs. Stores the latest state of the data.
Gold – Curated, well-structured data compliant with the organizations BI-tools and AI-algorithms. Moreover, this zone could also be used for feeding a traditional Data Warehouse during a transition period.

Visualize – Exposes the data in a way that make sense to the end-users through reports and dashboards

Features and Tools – Related to Microsoft and their investments to provide a seamless analytics experience for Dynamics 365 customers. Examples being the native service for automatically exporting Dynamics 365 data to the Data Lake in near-real time. Another example is the CDMUtil-tool for materializing views based on Dynamics 365 data using Synapse Analytics according and the Common Data Model schema.

The data lakehouse pattern is a relatively new concept in the data analytics scene. It was first introduced in 2017 in relation to the Snowflake platform. In 2020 the term became widely known when adopted by Databricks for its Delta Lake platform.

Benefits and Limitations with the Data Lakehouse patters

As always there are tradeoffs to be made when choosing between different BI- and Analytics patterns. Below is a list of examples based on our experience.

Benefits

Cost efficient cloud storage solution for Big Data scenarios
Flexible solution since structured, semi-structured and un-structured data could be managed
Strong support for Data science and Machine Learning experimentation

Limitations

Risk of creating a “data swamp” rather than a Data Lake if proper governance is not established
Good fit for cloud environments but not for on-prem due to main concept of separating compute and storage
Considered as a relatively new (immature) technology and less adopted by organizations which could require additional training and technology assessment activities

Success factors during implementation

Whether you want to modernize legacy systems to support advanced analytics scenarios or build a new BI- and Analytics solution from scratch, there are key considerations to be evaluated.

First, ensure the right mix of competence. Dynamics 365 specialists for business process- and data model understanding including how to configure the native export of data to the Data Lake. BI- and Analytics specialist to realize your target architecture. If using partners, this typically involves onboarding separate niche players for Dynamics 365- and Analytics competence.

Secondly, familiarize yourself with the core concepts and available standard capability. Preferably, by executing a Proof-of-Concept in a Sandbox environment where a simple hypothesis is validated. For example, export Dynamics 365 data to the lake and then apply a compute service such as Apache Spark by creating a Notebook to curate the data and then finally visualize insights in a report using Power BI.

Finally, ask yourself why you need a data storage solution? Is it for regulatory reporting, business intelligence, real-time analytics and/or data science? Ensure that your analytics needs are well defined to ensure you are targeting the correct personas across your organization. Considering Dynamics 365 F&SCM, the workload ships with a vast number of reporting and BI-capabilities to cater for various needs. Exporting the data to a data lake for further processing is only one available option. Understanding which capabilities that is available out-of-the-box will ensure you maximize the value of your existing Dynamics 365 investments.

Would you like to know more, contact us at: contact@engagenow.com

Dynamics 365 Commerce technology modernization

April 6, 2022 by Tobias Lång

The in-store experience is fundamentally changing as a consequence of new customer demands. Microsoft is therefore performing technology modernization across the Dynamics 365 Commerce workload such as releasing a new Store Commerce Application that will significantly improve the IT Operations- and End-User experience.

The new Store Commerce App is primarily a technology upgrade which means from a capability- and end-user perspective, the difference is limited. However, from a technology standpoint there are major improvements related to performance, reliability, and long-term support on the Windows platform. The biggest difference being a new Commerce Software Development Kit (SDK) with the objective of reducing operational overhead and maintenance cost.

Store Commerce App

As of end of March 2022, the new Store Commerce App is General Available on version 10.0.25. This means a downloadable package is now available for download in Lifecycle Services (LCS) portal for all existing customers.

Initially, the new Store Commerce App is only available on Windows devices but with a clear roadmap from Microsoft to provide Android/iOS availability by Fall 2022/Spring 2023.

Dynamics 365 Store Commerce App

At Engage Group, we are committed to supporting our customers throughout their Dynamics 365 Commerce journeys and have already extensive experience with the new Store Commerce App. While the Preview- versions struggled with certain bugs related to the installation experience, we have now successfully been able to install the GA version without any issues at internal environments as well as at customer Sandbox environments.

With the availability of the new Store Commerce App, it is time for customers currently running the legacy Moder POS (MPOS) to start preparing for a transition to the new app experience.

Call to action

Microsoft has published a high-level release timeline for the new Store Commerce App across various platforms (see table below). Most importantly, the table indicates End-of-Life support for exiting MPOS by Spring 2023 although Microsoft recommends adopting the new technology modernizations in calendar year 2022 to take advantage of all the improvements. Hence, customers’ needs to start planning accordingly to ensure a smooth transition.

MPOS End-of-Life support – 10.0.32 (Spring 2023)
Retail SDK End-of-Life support – 10.0.32 (Spring 2023)
Android/iOS sealed installer available – 10.0.29 (Fall 2022)

Source: Microsoft Whitepaper: Modernizing the Dynamics 365 Commerce in-store technology stack

Arguments in favor of starting early with the transition to the Store Commerce App:

Parallel rollout – Given the dependencies between the Store Commerce app and the new Commerce SDK, requiring re-activation of MPOS or Store Commer app once rolled out, Microsoft strongly recommends planning for rolling out both changes in parallel. This requires thoroughly planning to not affect existing business.
Custom footprint – Migrating to the new Commerce SDK might require one-time code changes to existing customizations. Hence, early testing is crucial to identify any breaking changes at an early stage.
Business value – In the “Modernizing the Dynamics 365 Commerce in-store technology stack”- whitepaper, Microsoft highlights a number of value drivers for transitioning to the new Store Commerce App and Commerce SDK. The sooner you are ready, the sooner you could start leveraging the new capabilities to start driving business value across your organization and customer base.

There are numerous alternatives available for approaching a technology modernization such as this. At Engage Group, we are committed to adopting agile principles where the “Proof of Concept”- model has proven highly successful for similar engagements. Independent of approach, it is of highest importance to start planning your transition together with your product team and any external partner or specialist if applicable. Activities to be included but not limited to the bullets below are:

Create a high-level project plan and ensure alignment with Microsoft release plan
Highlight dependencies towards related initiatives (e.g., One Version considerations or other Backlog refinement activities)
Determine current MPOS custom footprint to assess the need for one-time code changes for existing customizations.
Decide upon Environment Strategy for deploying and testing the new Store Commerce App in a Sandbox environment.
Evaluate peripheral equipment that needs to be tested by the new Store Commerce App
Evaluate external Payment Solution Provider (PSP) solutions if applicable that needs to be tested by the new Store Commerce App and Commerce SDK
Determine device strategy for adopting the Store Commerce App (e.g., Windows platform only and/or Android/iOS).

Useful links

Below is a number of useful links to valuable Microsoft resources to help you get started with the new Commerce SDK and Store Commerce App planning.

Migrate to the Commerce SDK – Commerce | Dynamics 365 | Microsoft Docs

Commerce Extensions Part 3: Migrate POS Extension to the new Independent POS SDK | December 8, 2021 – Microsoft Dynamics Blog

Dynamics 365 Commerce Forum – Support & Help

GitHub – microsoft/Dynamics365Commerce.InStore: Repository for hosting the Dynamics 365 Commerce in store Samples

Contact Engage Group for advisory

Are you in need of advisory to plan for you transition to the new Store Commerce App including the Commerce SDK? Or are you curious to tap into Engage Group’s vast experience on the topic, do not hesitate to contact us!

Agile for ERP

March 21, 2022 by Engage Group

Finally, the agile mindset and approach has reached the more conservative ERP world. At Engage, we have been involved in several such projects where it has been proven that it not only works but also has great advantages over the traditional waterfall method of ERP implementation.

We do see strong advantages in adapting to agile principles, ways-of-working and tools:

• Increase productivity
• Improve collaboration (both IT and Business)
• Increase transparency to priorities and backlog
• Decrease risks

“Waterfall vs Agile” is not always helpful as a “one or the other“ distinction in the new cloud world but we do find some key items to be essential for success:

• Choose the best team for the task – easier said than done if you are in the hands of an outsourcing partner
• If possible, have a Minimum Viable Product approach
• Make sure to have a process for handling changed and new priorities
• Early and repeated validation of the solution to make sure that processes and scenarios meet the business needs and requirements
• Make sure to get ambassadors among the business stakeholders to gain confidence, experience and increase engagement

Organizations that are the most successful in embracing agile, understand that agile methodologies are not something that organizations do, but rather something that the organization becomes. It takes a long time to become agile, but the payoff is proven even in the ERP-world for application lifecycle management.

Architecture Board + Lean Coffee @ Engage

December 28, 2021 by Engage Group

How do you ensure structured and meaningful meetings with high participants involvement without requiring extensive preparations? At Engage Group we have adopted a “Lean Coffee”- approach for our reoccurring Architecture Board meetings.

Lean Coffee

The idea of “Lean Coffee” as a meeting concept was born in Seattle in 2009. Founders Jim Benson and Jeremy Lightsmith wanted to create a simple, yet efficient meeting format. The emphasize should be on providing “the least structure necessary for a coherent and productive meeting. No more, no less”. – Leancoffee.org

How does Lean Coffee work?

Set up a Kanban Board – Create a Kanban board to provide a structure for the conversation
Decide what to discuss – Add topics for conversation into the Kanban board
Vote and Talk – Vote for topics to discuss

Architecture Board at Engage

At Engage Group we have adopted the “Lean Coffee” concept to facilitate our recurring Architecture Board meetings. Considering the key concept of providing a structured approach, yet with minimal preparation, it allows us to have meaningful discussions with our team of highly experienced Dynamics 365 Solution- & Technical architects that is typically very busy in various customer engagements.

Due to the pandemic and geographical challenges of being located at different physical locations, we run these gatherings as virtual meetings where we take advantage of virtual collaboration tools such as Miro.

We start by spending a few minutes on brainstorming and adding topics to discuss using a virtual Kanban bord and sticky notes.

When the number of topics start reaching a certain point, we start voting on topics to discuss (each participant gets two votes).

Finally, we start discussing the highest voted topic. The sticky note is then moved to the “Discussion” column. Typically, we spend 10 minutes per topic with the possibility to extend by 5 minutes depending on need. When completed, we move the sticky note to “Discusses” and then continue with the next topic.

Actions derived from the various discussions are added into a separate “Actions”- column for follow-up purposes. See example below:

Picture: Example Kanban Board at Engage Group Architecture Board meeting

Conclusion

The Architecture Board is a vital part of Engage Group’s philosophy of creating a “learning first culture” where we continuously share our experiences and challenge way of working. We believe this creates an inspiring work environment which ultimately is beneficial to our clients.

What is you experience of using an Architecture Board in a Dynamics 365 context? If you are interested to learn mot about the Architecture Board at Engage, do not hesitate to reach out to us.

Get started with automatic testing

March 11, 2021 by Engage Group

Regardless of company or industry, there is a continuous need for innovation and the use of different types of software. Thus, there is also the requirement for efficient and well-functioning application testing.

This has been seen as a given in large parts general software development and new innovations have long been built with the help of automated tests and continuous deployments. Why has automation not come further for critical ERP systems?

One reason is that ERP projects have used and are still using classic waterfall projects where speed has not been prioritized. A false (to some extent) security has been built up with few major releases instead of a more agile way of working, that put higher demands on automated program testing and a faster deployment rate. Another historical factor is that the hurdle to get started has been too high and the projects have built a technical debt that feels too heavy and costly to overcome. The suppliers’ strategies in the area have simply also been too weak. Thankfully, large steps have been taken in the right direction in recent years.

Take these steps to get started:

Secure support: Make sure the management thinks this is a brilliant idea and long-term necessity.
Get help: Get help from an expert and make sure you get started! Start by setting a reasonable initial scope, define a clear set of rules, appoint and educate those responsible, and choose the right testing tool.
Get started: Start with simple processes and build from there. Be sure to inspire and educate along the way so more key people realize the importance and benefits of automating.
Improve continuously: Everything should not be automated – you will continue to have prototype-, user- and certain function-testing that there is no point in automating. Get started first and improve gradually. Learning by doing.

Feel free to contact us if you want to talk about automated testing in general or Microsoft Dynamics 365 testing specifically. At Engage, we have a methodology that we have used to help several customers get started, carefully selected tools, and set the scope for automated testing of Microsoft’s monthly updates or our own functional releases. The whole idea is based on getting started quickly with implementation in order to gradually build the customer’s own ability to drive testing.

Microsoft Ignite 2021 – Key takeaways from a BizApps perspective

March 9, 2021 by Tobias Lång

During the first week of March 2021, Microsoft Ignite was held as a fully virtual event uniting more than 100.000 IT professionals across the globe. Engage Group attended the event and will be hosting a webinar on the 18th of March to share our in-dept insights, see registration link HERE. Although Ignite covers the entire Microsoft Cloud offering, this article will focus on key takeaways from a Business Applications perspective. For a complete overview of all announcements, see: MS Ignite – Book of News 2021.

Key Takeaways

More insights related to Dynamics 365- and Power Platform was certainly expected. However, as the Microsoft Business Application Launch event being just 1 month away, it might be a thoughtful move by Microsoft to save the highlights for this event. With that said, there were still a few really exciting announcements.

Dynamics 365

A number of new- and updated capabilities were announced during the event which could be summarized into the following three pillars:

Dynamics 365 Intelligent Order Management – A completely new Dynamics 365 app that caught many by surprise. The app is designed to realize omnichannel scenarios such as “buy anywhere, fulfill anywhere”. To simplify, the idea is to connect sales channels with fulfillment options and then apply AI to surface exceptions and then provide recommended solutions.
Customer journey orchestration – An improved capability included in the Dynamics 365 Marketing app with the intention to create real-time personalized customer experiences for each individual. Also here, AI is utilized to drive personalization across digital and physical touchpoints.
New integrations between Dynamics 365 and Microsoft Teams – Enhanced workflow capabilities as well as communication enhancements tailored for Sales, Marketing, Customer Service, Human Resource and Commerce. Examples being shared Teams chat windows embedded into Dynamics 365 Sales and Tasks created in Teams which could be surfaced within the Dynamics POS client.

Summary: Microsoft continues to invest in enhancing collaboration across various Dynamics 365 apps and related services allowing customers to leverage new innovation in a faster pace than ever. Infusing AI capabilities across the Dynamics 365 ecosystem is the new normal. We have only seen the beginning here but make no mistake, AI will be a natural part of all CRM and ERP capabilities in the near future. Dataverse, the cloud storage solution which originated from the customer engagement apps is getting an increasingly dominant role within the Dynamics 365 and Power Platform landscape. A major driver is the new AI capabilities, which typically leverage Dataverse as a key component.

Power Platform

A new programming language together with a number of IT governance improvements summarizes the key announcements related to Power Platform.

Microsoft Power Fx – The first open-source, low-code programming language applicable across the entire Power Platform. This is not a new concept within Power Platform but has now received an official name. It still leverages Excel formulas to reduce barriers for citizen developers.
IT Governance and Security features – New features added in Power Platform admin center to efficiently monitor and secure data across large instances of Power Platform environments.

Summary: As low-code-/no-code adoption surges due to an increasing demand of digitizing manual processes, the need for robust governance will be increasingly important. Balancing user innovation but at the same time providing guardrails to secure sensitive data will be crucial. Customers has raised this as a concern within Power Platform throughout the years and Microsoft is now heavily investing in this area. Another interesting topic that was debated extensively was the idea of viewing low-code development as a “team sport” where business users and professional developers work together as “fusion teams”. The idea that the professional developer is no longer needed in a low-code environment is a common misunderstanding. Rather, the professional developer will continue to play a key role in terms of providing “building blocks” to business users which then could be utilized to solve complex scenarios.

For an indepth discussion on key takeaways from Microsoft Ignite 2021, make sure to register for our webinar the 18th of March.