Welcome back to our integrations blog series!

In the last post, we tackled authentication and all the other steps needed to set up a connection between Productive and Xero.

Now, let’s talk about data mapping – why do we need it?

As you already know, every system stores data differently, and without proper mapping, it’s like trying to fit together puzzle pieces from different sets.

Come along as we dig into the significance of data mapping in ensuring a seamless integration!

All data from Productive must be correctly mapped in Xero when copying invoices. We utilize explicit data mapping to ensure this.

The integrations model is used to implement all the integrations available in Productive, and each of these integrations is specific, requiring the mapping of different data uniquely. For this reason, the integration model defines an options attribute in JSON format, intended to store all data specific to a particular type of integration.

After successful authorization, the user explicitly selects integration settings (such as exporting invoice numbers or payment sync) and maps data (account code mappings) between systems.

Data such as taxes and currencies are retrieved from Xero and automatically stored in the integration options field. This information will be essential for the user in subsequent integration tasks, such as exporting invoices:

General settings:

When the customer is successfully connected, a modal with settings opens. Here, the customer decides how to export invoices to Xero from Productive and whether they want data synchronization between Xero and Productive. When creating a new invoice, invoice numbers are generated for the customer, e.g., Invoice – 1, which can be done through Productive or Xero. Similarly, a decision is made for the system that will generate the invoice PDF in its specific way, including the invoice number. Also, an initial status is set before creating any invoice related to one particular branch, which is sent as a reference field and internal note.

Map settings:

When setting up a Xero account, there is an option to map each service type. In other words, a specific Xero account is set for each type. The Xero account is equivalent to the Productive service type. Each line item on the invoice belongs to a specific service type. When copying an invoice to Xero, we want to know to which values we will map our values. This is the next step after configuring the general settings.

In this example, we can see what the object looks like for mapping options for the entire integration:

Let’s focus on the fieldOptions and fieldOptionLabelMap. With these properties, we define how we want to map service types and how we want to display them to the customer for selection. We obtain the list of Xero accounts from the integration model on the backend, and on the frontend, we map them using these getters:

To ensure the validity of the integration update form, a default Xero account MUST be selected. This is allowed if customers have multiple service types and want to map only some of them to specific Xero accounts. This also means the default selected Xero account will be used for other service types.

In conclusion, integrations are crucial in enhancing productivity and efficiency in tools like Productive. The post series highlights the significance of integrating with platforms like Xero, showcasing the OAuth 2.0 protocol for secure authentication and the meticulous data mapping process. This seamless integration not only simplifies tasks such as invoicing and payment synchronization but also ensures consistency and customization in transferring data between systems with distinct business logic.

In conclusion, integrations are crucial in enhancing productivity and efficiency in tools like Productive. The post series highlights the significance of integrating with platforms like Xero, showcasing the OAuth 2.0 protocol for secure authentication and the meticulous data mapping process. This seamless integration not only simplifies tasks such as invoicing and payment synchronization but also ensures consistency and customization in transferring data between systems with distinct business logic.

Integrations are a big part of any software, but they’re crucial to the functionality of a tool like Productive.

That’s why we decided to dedicate a post series to discussing the importance of integrations in Productive. From enhancing functionality by connecting different tools to streamlining processes and improving user experience, integrations make life easier. We’ll showcase one of our most-used integrations, the one between Productive and Xero, which allows users to seamlessly transfer invoices and synchronize payments between the two tools.

Your agency uses Productive for managing client projects and project financials. At month’s end, you face the daunting task of issuing and sending invoices upon invoices, then replicating the process in Xero. Afterward, you also have to individually mark each payment as received.

And this is where the power of integrations kicks in. With one simple integration, invoices are synced between Xero and Productive, and any payment received in Xero is automatically recorded in Productive.

The integration between Productive and Xero looks interesting, right?

But before actually using the integration, we need to set it up first, and that’s the main focus of this blog post! We’re exploring the implementation of the OAuth 2.0 protocol.

Let’s dive right in!

OAuth 2.0 (Open Authorization 2.0) is an authentication protocol that is considered an industry standard.

Utilizing the OAuth protocol, Productive is granted access to the Xero account without accessing the user’s credentials.

Key features of the OAuth 2.0 protocol:

In the following steps, the authentication process is explained:

1. The user authorizes Productive, granting access to their own data on the Xero platform:

2. After successful authentication, the user is redirected back to the redirect_uri defined in the previous step with two values added as query params:

Ensuring the redirection of users back to the Productive app is a crucial aspect of the OAuth flow due to the sensitivity of the information contained in the redirect_uri.
To ensure the user’s redirection to the correct location, we have securely stored the redirect URI within the Xero app.

3. Exchanging the verification code (code) for an access token (access_token):

4. Retrieving the generated tokens:

access_token is used to call the Xero API, while the refresh_token is used to refresh the access token once it has expired.

5. Retrieving and selecting the tenant whose resources are being accessed:

Each user can have multiple Xero organizations (referred to as tenants).
Xero requires that the xero_tenant_id field is sent as a header param in every HTTP request.

The following code snippet shows the retrieval of all available tenants, from which the user later selects a tenant for current integration in the Marketplace settings:

In Productive, as well as in many other applications, there is a list of applications called the marketplace. In the marketplace, customers choose the integration they want to use. When the integration is selected, the connection process begins by clicking the “Connect app” button. The connection flow will be demonstrated using the example of Xero.

When establishing an integration, it’s generated in the system. However, to ensure its successful creation, it must align with the authentication requirements on the backend. All integrations require a redirect URI. This URI is used to redirect to Productive after connecting the integration to adjust the settings of each integration. After generating the URI, the integration creation process begins.

If the organization has branches or subsidiaries, you’ll first need to select the subsidiary (branch) for which you want to use Xero.

When creating an integration on the backend, authentication verification is performed, and if all requirements are met, the integration process begins.

During the integration connection, the customer is directed to the interface of the system they want to integrate with, where they also need to provide information to verify if the user is existing and valid. Once completed, the process returns to the marketplace via the redirect URI, initiating the setup of the integration.

After the customer is redirected to Productive, the data obtained from the external tool is set on the integration model. For example, Xero requires that there is always a code and org representing the code and organization to which the integration connects and exports data in Xero.

Once the integration model has been updated, a call to the backend occurs again to update the integration with new data.

After updating the model, if there are no errors, we set the parameters of the model to transition to the integration setup route, i.e., editing the integration settings.

With successful authentication and integration connection, we’re finishing the first part of the integration posts series. As described in this blog post, understanding OAuth 2.0 becomes not just a necessity but a powerful tool to enhance user experience, safeguard sensitive information, and foster a more trustworthy digital ecosystem. After successful authentication, due to settings related to the external system redirect, in this case, Xero, it brings us back to Productive to continue with further integration setup.

In our next post, we’ll break down data mapping and show you why it’s a must-know for smooth integrations. Don’t miss out!

The more you put things off, the harder it gets to actually do them. That goes for dependencies too. Sure, dependencies need to be updated, but that doesn’t make the task any less miserable.

Before I start explaining what a dependency manager is, let me explain what a dependency is. 

Dependencies are third-party libraries that developers add to their projects so their lives would be easier. What that means is that someone wrote a piece of code that other people can use. It also means that it needs to be frequently updated. In most web projects the list of those libraries is located in the package.json file. 

Now that you know what dependency means, I want to introduce you to the idea of dependency management. 

Did you ever get caught up with a lot of non-updated dependencies and when you began updating them, you realized you missed a lot of things? Some of those updates can be helpful, but some of them can make your life miserable. 

At Productive, as developers we encountered the same issue. Some of our dependencies were out-of-date by several major versions which can lead to real issues, such as deprecations. So we started to think of a way to keep everything up-to-date. 

Manually handling them was not an option anymore. We have several repositories just for front-end projects and it takes a lot of time to track all those dependencies with their release updates. So, we decided to implement a dependency management bot. 

A bot like that not only notifies you about new versions, but it notifies you if a certain package has vulnerabilities. It suggests changes by creating Pull Requests. You’re probably wondering how can dependencies have vulnerabilities now. Well, some of them are depending on other dependencies, or it may be that the developers just wrote some shady code.

One of the problems we confront as developers is “notification noise”. 
Nobody likes to see a swamp of notifications and nobody has time to read them one by one. Therefore, we were trying to pick a configurable manager to control the level of noise. A few of them came up in my search—such as Dependabot, Renovate, Snyk, Dependencies.io, Depfu, etc. 

Dependabot is popular, mostly because it’s a GitHub manager, but it’s not quite as configurable as our final choice (wait for it!). One of the great things about Dependabot is that it can be used in combination with other managers (like Renovate). After comparing them, Renovate won. 

Renovate is more configurable and what I mean by that is that we can choose the dependencies that will be checked by it. We can set up a schedule for when the notifications will arrive, we can divide packages into all sorts of different groups we need. If you don’t like the way your PR looks, as you probably guessed—you can edit it. With this manager, you can also set an assignee who will get notified, and much more.

Since we didn’t configure the manager to auto-merge new updates, we need to handle them ourselves. So, when it does its job, usually I handle most of the updates for the Productive app. If there are some dependencies which I’m not so familiar with, I create a task and assign the PR to my colleagues.

Because I read most of the Release notes, sometimes I stumble upon some useful information about how a new update will be helpful for someone and then I inform them because new updates may solve their current workarounds (read: hacky solutions).

When we start updating dependencies we approach them differently. For every major package, one PR is created. For non-major dependencies, there is one PR where all dependencies are handled because they are less likely to cause bigger problems. 

Of course, the person in charge sets his PR link in a task so if some errors are made because of it, it can be easily traceable. When the whole list of packages is up-to-date, the developer merges that PR. After merging, the Renovate bot will auto-update the created PRs and if there aren’t any other updates the bot will auto-close it.

The issues we wanted to solve:

The first problem I wanted to solve was reducing the noise to a minimum. Since the notifications are not wanted every day, I set a schedule for when we get those notifications. The schedule is set by the “schedule” property. The value of “schedule“ will represent the fixed times when the pull requests get generated.

Because there are different time zones, don’t forget to implement a timezone for your or your company zone (ours is Europe/Zagreb) using the “timezone” property with this list: IANA Time Zone.

Since I’m not the only one who should be warned about updates, I found the “assignees” property which will set assignees who will be notified. For this property, you can use GitHub’s username or email.

Although I am the main assignee for dependency updates on all of our front-end projects, I didn’t want to read each PR independently. I started to look at how dependencies can be grouped, sorted, or something like that. 

So I found “One array to rule them all” which saved me from a lot of buzzing noises on Friday , and it was the packageRules array. Each object in that array specifies which updates I want in each PR. In one PR it was necessary to receive updates only for major versions, the other was for patch and minor versions and it looks like this:

or

Each group can be named whatever you want and prioritized by prPriority. I wanted to assign some groups to other colleagues so I don’t have to deal with everything. Because I am lazy. Just kidding. 

Other colleagues had different assignments because they were more familiar with some specific libraries, so I used matchPackageNames and matchPackagePatterns. 

The matchPackagePatterns came very useful because you can use RegEx. I almost forgot to mention that for some of the dependencies, it wasn’t necessary for them to be on the list of updates. 

Why? Well, because there’s a Core team that deals with them every day at Productive, so I can set enabled: false to stop Renovate from notifying me or them. 

For example: 

Also, one of the useful commands is baseBranches. With that command, you set which branch will be checked by the bot. That can be useful because most software products are not updated daily and the companies have specific procedures until some feature goes out to the public.

Here’s an example of the renovate.json file with the basic configuration previously mentioned: 

For every other option you want to add or explore, feel free to read Renovate’s documentation.

Below, an example of one PR with a table (photo 1) and Release notes (photo 2).

The table is pretty self-explanatory. You can see which package or whatever you want to call it can be updated to which version with some other data. The table itself is configurable, so you can tailor it to your preferences.

Then there’s a section with the release notes in which you can inspect each update individually with their specific commits and proper descriptions.

Lastly, not everything is perfect. There were some issues we encountered in the implementation process. 

For instance, there are two types of PRs, normal and immortal ones. A normal PR is a PR with only one dependency. When you close a normal PR, the main problem is that the manager will “think” that you maybe do not want to use that version of dependency and it will not notify you again. Immortal PRs can be closed because the manager will reopen them again, in our case those are grouped PRs. So, how do we deal with that feature/issue? We don’t close PRs until each package is up-to-date.

Another issue I encountered is that PRs will be created inside a schedule, but the manager will notify you outside of a schedule when updates are made. For example, the PR schedule for creating PRs is on Friday, but in the meantime, a new version of a certain dependency would be published and I would receive a notification. That could be useful, but since I update versions on a weekly basis I added the :noUnscheduledUpdates property, which limits notifications only inside the schedule.

One issue that still bothers me is that I receive vulnerability warnings for dependencies in one PR for every warning. I wanted to group them, but it seems nobody figured out a solution for that, yet. Depending on when you read this article and if this problem is solved, please contact me. Our method is to update only the dependencies to stable versions and use that as a workaround.

The last issue we encountered was with the range strategy. Renovate will not update packages if there’s a range indicator such as ^. That was weird to me, but they have a solution for that—you can set a rangeStrategy and Renovate has various options so that you can adjust to your app as you wish. We solved it by adding a “bump” rangeStrategy. We started with “replace” but when the manager reopened an immortal PR, dependencies that were on the closed PR’s list wouldn’t show up.

In conclusion, if you ever get assigned to manage dependencies, for the love of God, use a dependency manager. If you have to ask why, please read this blog post again, or contact me to remove this post. 

But, for real, it just makes your life easier. Renovate is a solution that was best suitable for us. Because of it, I do not need to go to the NPM’s website to track every package if it has a new version available.

One of the annoying issues it also solved for me was that it keeps the noise to a minimum because of the “schedule” property. The noise was also reduced with the creation of groups for different types of versions.

Also, PRs are very easy to read and last, but not least, one of the best features of it is—it’s FREE.