Recipe: Implementing Flow reporting on Postgres

Boomi Flow has a reporting infrastructure, that lets us analyze, store, and query state data right from the comforts of our own database or analytics user interface. Flow reporting currently supports storing events, states and state values. A state is a running instance of a flow. State data provides information all running states.

Starting from scratch.. to implementing reporting

This recipe shows us how to enable reporting on our Flow tenant, create a player with the reporting mode on, install the Flow reporting app in Heroku, deploy a Postgres database, and access state data from the database.

No prior database knowledge is required to follow the recipe.. and the entire process should take us roughly an hour to implement. Now lets just sit back, relax, and start reading.

Tool set
  • Boomi Flow username/password. (Get a free fully-loaded Boomi Flow account here.)
  • A Heroku account. (Get a free Heroku account for testing here.)
  • The instructions show how to work with the open source pgAdmin tool to interact with the database. Any client will work! (Download pgAdmin from here.)

Concept

If we would like a green-room view to understand what is happening behind the scenes, and learn about a few extras, this section is for us. If we like, we can also jump straight to building the implementation; coming back to the concepts for a better understanding once are done with the implementationing (wait, that’s not an actual word).

Before we can access flow state data via an external endpoint, three things need to happen.

  1. We need to enable reporting in our tenant.
  2. We need to set a reporting endpoint.
  3. We need to choose the reporting mode we would like to use.

We can enable reporting from the Tenants tab in the drawing tool. This is also where we will add the reporting endpoint. The reporting endpoint URI is the HTTP endpoint that the Boomi Flow platform will post state updates to, in near realtime, when running a flow. This is set on a per-tenant basis. (The performance of this endpoint does not affect the experience in the running flow; state updates are posted asynchronously.) We also need to configure the reporting mode we would like.

There are four reporting modes we can choose from:

  1. null: Do not report any state data.
  2. PATH: Report only the information in the state that pertains to the path a user took through the flow. This includes location information (if provided), as well as which users interacted at which step.
  3. VALUES: Report only the values collected in the state as a user goes through the flow. The values are posted in JSON format.
  4. PATH_AND_VALUES: Report the information included in both the PATH and VALUES modes.

There are three ways we can configure the reporting mode in our tenant:

The reporting mode can be set in the Flow player (we can select this player while running one or many flow/s), in the Flow Run URL (for a single flow), or in the initialization request to the Run API (if we are using the API directly).

  1. Flow player – We can create a new player based on the default Flow player, and set the reporting mode by changing the value of the options.reportingMode property. (This recipe shows us how to do this.) The advantage of configuring the player is, once we have set the reporting mode, we can reuse the same player across multiple flows.
  2. Flow run URL – To set the reporting mode when running a single flow by the URL, we can add a reporting-mode query parameter, passing the desired mode from above as the value. For example: https://flow.manywho.com/07f799a4-af7c-449b-ba7c-f2f526f7000b/play/default?flow-id=b6b211f0-b344-4f67-b7b1-b8b62868b2a4&reporting-mode=PATH
  3. Using the Run API – If we are initializing a flow directly, by using the Run API, we can set the reporting mode in the request, by passing a value for the reportingMode property in the main request object’s root.

Let’s do this!

  1. Click here or copy-paste this link (https://github.com/manywho/reporting) to a browser to open the Boomi Flow Reporting repository in GitHub.
  2. Scroll down the page to the README.md document. There is a section on Heroku.
  3. We will perform a one-click deploy of the application. Yes, this is super convenient! All the grizzly app configurations bits are automatically done for us. Okay, let’s click the purple Deploy to Heroku button.

    This opens Heroku, with our own installation of the Boomi Flow Reporting app.
  4. Enter a name for the app (Say, boomiflowreporting) in the App name field.
  5. Scroll down the page and click Deploy app.

    It will take a few seconds for the app to build. When finished, we will get a message saying Your app was successfully deployed.
  6. Click Manage app.

    This opens the app Overview screen.
  7. Click Heroku Postgres.

    This opens the Overview screen for the database.
  8. Click Settings.
  9. Click View Credentials.

    This lists the credentials of the database we just created.

    We will use these credentials to create a new pgAdmin connection in a bit.
  10. Click boomiflowreporting. This opens a new tab with the Overview for our app.
  11. Click Open app.

    This opens a new tab with the app URL. Yes, we see a Not Found message – that is okay.
  12. Copy the URL.
  13. Let’s log in to our Boomi Flow account and open the tenant where we want to enable reporting.
  14. Click Tenant.

    This opens the Tenant tab.
  15. Scroll down the screen to where it says Reporting, and expand the Reporting options.
  16. Paste the Heroku app URL we copied earlier in the Reporting endpoint Uri field.
  17. We will now add a suffix to the URI. The suffix is needed if we are running the Flow reporting service. We can run our own reporting service as well, if we like. Add this to the reporting endpoint URI which we pasted: api/reporting/1/states. This is what the URI looks like now:
  18. Click Save Tenant.
  19. We will now create a new player and set the reporting mode. Click Players.

    This opens the Players tab.
  20. Click New.
  21. Enter a name for the player in the Name field. Say, FlowReporting.
  22. The player needs to be based on an existing player. Let’s select the default player.
  23. Click Save to save the player. We get a system message saying Player saved successfully.
  24. Close the message.
  25. Find the line of code in the player that says reportingMode: queryParameters[‘reporting-mode’], (Ln 134 in the image.)
  26. Delete queryParameters[‘reporting-mode’] and copy-paste PATH_AND_VALUES in its place. (Exactly like this; in upper case, with the underscores, and within single quotes. Do not forget the comma at the end!)
  27. Click Save to save the player.
  28. Open pgAdmin and click Add New Server.

    This opens the Create – Server dialog box.
  29. Let’s give our server a name. How about Art Vandelay? Kidding, kidding.. let’s call it Flow Reporting.
  30. Click Connection.

    We will now enter the Host name/address, Maintenance database, User name, and Password. As mentioned earlier, this information is available in the Heroku Database Credentials screen, available under Settings in our database.
  31. Click Save when done.
  32. The server is now ready and available. Expand Servers in the left-hand-side to bring up the Flow Reporting server.
  33. Click the +sign next to Flow Reporting to expand the options. This shows Databases, Login/Group Roles, and Tablespaces.
  34. Click the +sign next to Databases.

    This lists all the databases available in Heroku.
  35. We will need to find our database from the list. This is easier than it looks! The databases are arranged in alphabetical order.. and our database will not have the red cross icon on it. Let’s give it a try..

    Found!
  36. Click the +sign next to the database. This pulls up Casts, Catalogs, Event Triggers, Extensions, Foreign Data Wrappers, Languages, and Schemas.
  37. Click the +sign next to Schemas.
  38. Click the +sign next to Public.
  39. Right-click on Tables and select the Query Tool option.

    This is what the screen looks like now:

    Wondering if you will need to write a complicated query now? Nope. Boomi Flow has us covered!
  40. We have examples schemas available in the Boomi Flow Reporting repository. Click here or copy-paste this URL: https://github.com/manywho/reporting/tree/master/src/main/sql to a browser.
  41. Open the states.sql file.
  42. Copy the code.
  43. Let’s go back to pgAdmin and paste the code now.
  44. And.. Execute! In other words, let’s click the icon that looks like a flash of lightning.

    We get a Query returned successfully message.

    Tables accomplished!
  45. Right-click on Tables and select Refresh.
  46. Click the +sign next to Tables. We can see we have two tables now – states and statevalues.
  47. Time to reap the benefits of our labor! Right-click on states, selecting View Data and then View All Rows.

    This retrieves state information from our Flow tenant.

    Wait a sec.. our data output fields are empty! Why is that? Elementary, dear Watson. We are not running any flows in the tenant where we have reporting enabled.
  48. Let’s publish and then run a flow… remembering to select the FlowReporting player we had created earlier.
  49. Right-click on Tables in pgAdmin and select Refresh.
  50. Select states and click the executing lightning icon. And, oh, look! We have data now!

    How often is this information updated? Well.. every time as user does something, or we the Flow spinner — is a rough indicator. Why is there only one row though? Because we have only one instance of the app running.
  51. Let’s go ahead and run several apps in the tenant where we have reporting enabled.
  52. Right-click on Tables in pgAdmin again and select Refresh.
  53. Select states and click the executing lightning icon.

    Every row shows state data from one instance of an app. We can have thousands of users, interacting with thousands of instances of our app, and of course.. just a few thousand more rows. How cool is that!

Check out the glossary for a definition of terms and key concepts that appear in the Boomi Flow website, drawing tool, technical documentation, blogs, and marketing communications.

If you need to take a closer look, click on the images to enlarge them. Have a question? Click the Help button on the bottom right-hand corner of this page to ask.