Parts of a Synchronization Framework

Parts of a Synchronization Framework

In the last post, we talked about what are deltas and how by using them we can synchronize data structures.

So, in this post, I will describe the necessary parts needed to implement Delta-based synchronization, let’s start

  • Data Object: any database, object, graph, or file system that we are tracking for synchronization
  • Delta: a delta is a data difference between the original and the current state of a data object
  • Node: is a point of the synchronization network, there are 2 types of nodes
    1. Client node: a node that is able to produce and process deltas
    2. Server node: a node that is used only to exchange deltas, it can optionally process deltas too.
  • Delta Store: storage where you can save deltas so you can later exchange them with other nodes
  • Delta Processor: a utility that helps you includes the deltas created in other nodes in your current copy of the data object

Now let’s implement this concept to synchronize a database

Ok so each database that we want to synchronize will be a client node and a client node should be able to produce and store deltas and to process deltas produced by other nodes, so our database node should look like the following diagram

The server node can be as simple as just delta storage exposed in an HTTP API as you can see in the basic server node diagram or use several delta storages and delta processors as show on the complex server node diagram

                                         

 

And with those diagrams, I finish this post, in the next post we will implement those concepts in C# code

Data synchronization in a few words

Data synchronization in a few words

To Synchronize data is one of the most challenging tasks out there, especially if you are working with LOB applications

There are many ways to synchronize data, the most common technique is to compare records by modification date and then merge the data to create a master record.

Here the main problem is that you have to have a way to compare each type of record, so it gets cumbersome as soon as the number of types in your application begins to grow.

Also, you have to have a log of what gets created and what gets deleted so you can do the same changes on the other nodes

Delta-based synchronization

delta-based synchronization is a problem of identifying “what changed” between each execution of a sync process

A directed delta also called a change, is a sequence of (elementary) change operations which, when applied to one version V1, yields another version V2 (note the correspondence to transaction logs in databases). In computer implementations

In delta synchronization, there are 2 main tasks

  • Record the deltas (or small difference of data between nodes)
  • Transmit these differences to the other nodes so they can process them

Implementing Delta-based synchronization for relational databases

The schema above represents a blog’s database, it’s a really simple schema so its easy to understand, now this is the scenario

We have the main database that we will call “Master” and 2 other databases named client A and client B.

Now let insert data in the master

Each DML statement should be converted in a delta (or a data difference)

Δ1

Δ2

Δ3

Copy deltas Δ 1, Δ 2, Δ 3 to the clients

So, after processing the deltas on each client, the data in all databases should look like the picture below

 

 

So that’s it for this post, in the next post we will be examing each part that is necessary to do delta-based synchronization

 

 

 

How to fix “The type initializer for ‘Gdip’ threw an exception” caused by the netcore framework depencency,  when you run a Xaf Blazor App on ubuntu linux 18.04

How to fix “The type initializer for ‘Gdip’ threw an exception” caused by the netcore framework depencency, when you run a Xaf Blazor App on ubuntu linux 18.04

If you are running Xaf Blazor in ubuntu 18.04 you might have seen the following exception

The type initializer for ‘Gdip’ threw an exception.
at DevExpress.ExpressApp.Actions.ActionBase.OnHandleException(Exception e) at DevExpress.ExpressApp.Actions.ActionBase.ExecuteCore(Delegate handler, ActionBaseEventArgs eventArgs) at DevExpress.ExpressApp.Actions.PopupWindowShowAction.DoExecute(Window window) at DevExpress.ExpressApp.Actions.PopupWindowShowAction.DialogController_Accepting(Object sender, DialogControllerAcceptingEventArgs e) at DevExpress.ExpressApp.SystemModule.DialogController.Accept(SimpleActionExecuteEventArgs args) at DevExpress.ExpressApp.SystemModule.DialogController.acceptAction_OnExecute(Object sender, SimpleActionExecuteEventArgs e) at DevExpress.ExpressApp.Actions.SimpleAction.RaiseExecute(ActionBaseEventArgs eventArgs) at DevExpress.ExpressApp.Actions.ActionBase.ExecuteCore(Delegate handler, ActionBaseEventArgs eventArgs)

The error is caused by missing dependency, so the DotNet runtime itself will throw that exception. Also, I want to highlight that the exception is not related to XAF, you can read more about this problem here https://github.com/dotnet/runtime/issues/27200

To get the missing dependency just open a console and run the following commands

sudo apt-get update -y

sudo apt-get install -y libgdiplus

 

 

 

 

 

How to monitor your aspnet core application on Ubuntu Linux

How to monitor your aspnet core application on Ubuntu Linux

Here are some recommendations to host your new shiny aspnet core app on Linux in this case in Ubuntu 18.04

First, create a user with the name aspnetapp

sudo adduser myaspnetapp

 

after executing the command, you will have a new folder in your home directory the folder will have the same name as your username so in this case “myaspnetapp”

now let’s SSH to with the new user you just created you can do that using your favorite SSH client, for example, if you are using windows you can use putty

when you log in with the new user you will be in its home folder, now we can create a folder called app with the following command

 

mkdir app

your folder structure should look like this now

/home/myaspnetapp/app

Now we are ready to upload the files. By now should have already compiled and publish your application to run in Linux, if you have not done that yet then you should take a look to this article https://www.jocheojeda.com/2019/06/10/how-to-create-a-self-contained-netcore-3-console-application/

There are many options to upload a zip file but I think is the best way is to use the secure copy command from linux “scp”, I won’t explain how you should call the scp command but if you are using windows you can run that command from the WSL console and if you are using Linux the command is already built-in, anyway here is an article about it https://linuxize.com/post/how-to-use-scp-command-to-securely-transfer-files/

Here I will write an example of how the scp command should look like and you adjust it to your needs

scp publish.zip myaspnetapp@200.35.15.25:/home/myaspnetapp/app

so that command above will do the following, it will copy the file publish.zip from the local folder to a server running on the following the IP 200.35.15.25 and the folder “/home/myaspnetapp/app”

now let’s unzip the content of the folder zip with the following command

unzip publish.zip

 

What we have done so far:

  1. We have created a user in the OS
  2. We have created a folder to host our application within the user home folder
  3. We have uploaded a zip file containing our application the folder “/home/myaspnetapp/app”

 

Now that the app is already in the server we need to change the permission of the folder where the app lives to 0777, you can learn more about Linux file system permissions here https://www.guru99.com/file-permissions.html

Creating a service to monitor the app

The next step to monitor our app is to use systemd is an init system that provides many powerful features for starting, stopping, and managing processes.

Let’s start by creating a service file in the following path “/etc/systemd/system/”

You can do that with the following command:

sudo nano /etc/systemd/system/MyExecutableFile.service

 

here is how the content of the file should look like

 

[Unit]

Description=A description of your app


[Service]

WorkingDirectory=/home/myaspnetapp /app

ExecStart= /home/ myaspnetapp /app/MyExecutableFile

Restart=always

# Restart service after 10 seconds if the dotnet service crashes:

RestartSec=10

KillSignal=SIGINT

SyslogIdentifier= MyExecutableFile

User=apache

Environment=ASPNETCORE_ENVIRONMENT=Production


[Install]

WantedBy=multi-user.target

 

Here is a little explanation of what you might need to change if the file above

WorkingDirectory: this is the working directory, usually is the same where the app lives

ExecStart: This is the executable file how what will you write here will depend on your application if its self-contained you just need to write the name the full path of the executable file otherwise you need to write the path to the dotnet runtime and then the path to your dll as show below:

/usr/local/bin/dotnet /var/www/helloapp/helloapp.dll

RestartSec: this is the time to wait before trying to restart the app after if the process crashes

SyslogIdentifier: the app identifier for sys logs

User: this is really important since the app will run under this user privileges, so you need to make sure that the user exists and that is able to access the files needed to start the app

That is all that we need for the service file now we need to go back to the console and enable our new service, you can do that with the following command

sudo systemctl enable MyExecutableFile.service

To start and stop the service you can use the following commands

//To Start
sudo systemctl start MyExecutableFile.service

//To Stop
sudo systemctl status MyExecutableFile.service

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Install TestCafe in Ubuntu 18.04

Install TestCafe in Ubuntu 18.04

Lately, I have been playing a lot with TestCafe which is a testing tool for website, there are 2 parts of TestCafe, the first part is TestCafe studio which is a tests editor tool where you can create or edit new tests that will be eventually used in the TestCafe test runner, both TestCafe and the test runner runs on Windows, MacOS, and Linux.

So what we are going to do today is to install TestCafe on Ubuntu Linux 18.04, in this tutorial we will only install the test runner because, in the end, my main goal is to use TestCafe to emulate the load on a server, so let’s get started

First, we need to install NodeJS and NPM, this is kind of tricky depending on your OS version, for example, if you run this command :

sudo apt -y install nodejs

in Ubunto 18.04 you will end up installing NodeJS version 8 something… that’s too old for what we need to do, so first let’s install the latest version of NodeJS, in this case, this is version 12

Installing NodeJS and NPM

1) First, let’s update our repository information

sudo apt update
sudo apt -y upgrade

2) Manually add Node.js APT Repository


sudo apt update
sudo apt -y install curl dirmngr apt-transport-https lsb-release ca-certificates
curl -sL https://deb.nodesource.com/setup_12.x | sudo -E bash -

3) now let’s install NodeJS

sudo apt -y install nodejs

4) This step is optional, if you want to verify the versions of NodeJS and NPM you can use the following commands:

node --version
npm --version

 

Installing TestCafe

To install TestCafe you only need to execute one command, so here it is

npm install -g testcafe

That was easy !!!

That is everything for this post, see you!!!

 

 

 

Ubunto add missing resolution (18.04.4 LTS && 20.04 LTS)

Ubunto add missing resolution (18.04.4 LTS && 20.04 LTS)

1) Determine Name of Display

sudo xrandr -q

Determine Name of Display

 

In my case, I will add the missing resolution to the display HDMI-2 (below you can see that I highlight all the current resolutions for this display)

 

List of supported resolutions

 List of currently supported resolutions

2) Run command to calculate VESA CVT mode lines by given resolution

cvt 2560 1080 (in this case 2560 is my monitor width and 1080 is the hight, you should replace these values to something that match your current need)

3) Now we need to run the command “sudo xrandr –newmode” and then paste the output value from step 2, in the end, it will look like the snippet below

xrandr --newmode "2560x1080_60.00"  230.00  2560 2720 2992 3424  1080 1083 1093 1120 -hsync +vsync

 

Add new resolution

4) Now add the newly created mode for your display device, the structure of the command is “xrandr –addmode DisplayId Resolution”, you should replace the DisplayId for the id you got in step 1 and the resolution from what you got in

xrandr –addmode HDMI-2 “2560x1080_60.00”

Add New Resolution To Display

Add New Resolution To Display

 

5) Set the new resolution

Set The New Resolution

Set The New Resolution

6) So far we added the new resolution and associated it to the monitor that we want, this is not permanent it will just work until you reboot, so to make these changes permanent you need to edit your profile using the following command

gedit ~/.profile

 

Paste the commands from step 3 and 4 and save the file

 

Edit Profile To Add Resolution

Edit Profile To Add Resolution

How to delete all temporary files from (bin,obj,packages) your solution folder

How to delete all temporary files from (bin,obj,packages) your solution folder

Sometimes you want to send a zip with your visual studio solution or project, but after you archive your files you realize that the file size is huge, well worry no more you can use the following script (a .bat file) to clear all the temporary files in your visual studio solution

just save the following code in a ms-dos bat file, name the file ClearTeam.bat

rem start of the script
@echo off

for /d /r . %%d in (bin,obj,packages) do @if exist "%%d" rd /s/q "%%d"
FOR /R %%H IN (*.log) DO del "%%H"
FOR /r %%G IN (*.bak) DO del "%%G"
FOR /R %%J IN (*.suo) DO del "%%J"

IF EXIST .vs rd .vs /s/q
rem end of the script

to run it, just double click in the bat file or execute it but call the file name in console, in this case, ClearTeam.bat

 

 

 

 

Replacing WCF with AspNetCore Rest API as transport layer for XPO

Replacing WCF with AspNetCore Rest API as transport layer for XPO

I have been using XPO from DevExpress since day one. For me is the best O.R.M in the dot net world, so when I got the news that XPO was going to be free of charge I was really happy because that means I can use it in every project without adding cost for my customers.

Nowadays all my customer needs some type of mobile development, so I have decided to master the combination of XPO and Xamarin

Now there is a problem when using XPO and Xamarin and that is the network topology, database connections are no designed for WAN networks.

Let’s take MS SQL server as an example, here are the supported communication protocols

  • TCP/IP.
  • Named Pipes

To quote what Microsoft web site said about using the protocols above in a WAN network

https://docs.microsoft.com/en-us/sql/tools/configuration-manager/choosing-a-network-protocol?view=sql-server-2014

Named Pipes vs. TCP/IP Sockets

In a fast-local area network (LAN) environment, Transmission Control Protocol/Internet Protocol (TCP/IP) Sockets and Named Pipes clients are comparable with regard to performance. However, the performance difference between the TCP/IP Sockets and Named Pipes clients becomes apparent with slower networks, such as across wide area networks (WANs) or dial-up networks. This is because of the different ways the interprocess communication (IPC) mechanisms communicate between peers.”

So, what other options do we have? Well if you are using the full DotNet framework you can use WCF.

So, it looks like WCF is the solution here since is mature and robust communication framework but there is a problem, the implementation of WCF for mono touch (Xamarin iOS) and mono droid (Xamarin Android)

You can read about Xamarin limitations in the following links

Android: https://docs.microsoft.com/en-us/xamarin/android/internals/limitations

iOS: https://docs.microsoft.com/en-us/xamarin/ios/internals/limitations

I don’t want to go into details about how the limitation of each platform affects XPO and WCF but basically the main limitation is the ability to use reflection and emit new code which is needed to generate the WCF client, also in WCF there are problems in the serialization behaviors.

Well now that we know the problem is time to talk about the solution. As you know XPO has a layered architecture ( you can read about that here https://www.jocheojeda.com/2018/10/01/xpo-post-5-layered-architecture/)

So basically, what we need to do is to replace the WCF layer with some other technology to communicate to the database server

The technology I’ve selected for this AspNetCore which I would say is a really nice technology that is modern, multi-platform and easy to use. Here below you can see what is the architecture of the solution

AspNetCore

Rest API

So, what we need basically is to be able to communicate the data layer with the data store through a network architecture.

The network architecture that I have chosen is a rest API which is one of the strong fronts of AspNetCore. The rest API will work as the server that forward the communication from XPO to the Database and vice versa, you can find a project template of the server implementation here https://www.jocheojeda.com/download/560/ this implementation references one nuget where I have written the communication code, you can fine the nuget here https://nuget.bitframeworks.com/feeds/main/BIT.Xpo.AgnosticDataStore.Server/19.1.5.1

Also we need a client that is able to interpret the information from the rest API and feed XPO, for that I have created a special client you can find here https://nuget.bitframeworks.com/feeds/main/BIT.Xpo.AgnosticDataStore.Client/19.1.5.1

The client implementation has been tested in the following platforms

  • Xamarin Android
  • Xamarin iOS
  • Xamarin WPF
  • DotNetCore
  • DotNetFramework

The client implementation has been tested in the following operative systems

  • Android 5 to 9
  • iOS 9 to 11
  • MacOS: Sierra to Catalina
  • Windows 10

In this link, you can see a full implementation of the server and the clients (XAF and Xamarin)

What is next? Well here are a few topics for the upcoming posts

  • Understanding JWT tokens
  • How to secure your data store service with a JWT token
  • Hosting multiple data store with a single service
  • Implementing your own authentication method
  • Examples examples examples

 

How to fix Error MSB4044 The “FilterAssemblies” task was not given a value for the required parameter “DesignTimeBuild”

How to fix Error MSB4044 The “FilterAssemblies” task was not given a value for the required parameter “DesignTimeBuild”

If you updated your visual studio to version 16.2.0 and you are using MSBuildSdkExtras your project won’t compile.

so to fix this problem (this is a temporary fix). You need to add the following lines to your csproj

<PropertyGroup Label="Android DesignTimeBuild error workaround"> 
  <DesignTimeBuild>false</DesignTimeBuild>
</PropertyGroup>