In the last post, we talked about what are deltas and how by using them we can synchronize data structures.
So, in this post, I will describe the necessary parts needed to implement Delta-based synchronization, let’s start
Data Object: any database, object, graph, or file system that we are tracking for synchronization
Delta: a delta is a data difference between the original and the current state of a data object
Node: is a point of the synchronization network, there are 2 types of nodes
Client node: a node that is able to produce and process deltas
Server node: a node that is used only to exchange deltas, it can optionally process deltas too.
Delta Store: storage where you can save deltas so you can later exchange them with other nodes
Delta Processor: a utility that helps you includes the deltas created in other nodes in your current copy of the data object
Now let’s implement this concept to synchronize a database
Ok so each database that we want to synchronize will be a client node and a client node should be able to produce and store deltas and to process deltas produced by other nodes, so our database node should look like the following diagram
The server node can be as simple as just delta storage exposed in an HTTP API as you can see in the basic server node diagram or use several delta storages and delta processors as show on the complex server node diagram
And with those diagrams, I finish this post, in the next post we will implement those concepts in C# code
To Synchronize data is one of the most challenging tasks out there, especially if you are working with LOB applications
There are many ways to synchronize data, the most common technique is to compare records by modification date and then merge the data to create a master record.
Here the main problem is that you have to have a way to compare each type of record, so it gets cumbersome as soon as the number of types in your application begins to grow.
Also, you have to have a log of what gets created and what gets deleted so you can do the same changes on the other nodes
Delta-based synchronization
delta-based synchronization is a problem of identifying “what changed” between each execution of a sync process
A directed delta also called a change, is a sequence of (elementary) change operations which, when applied to one version V1, yields another version V2 (note the correspondence to transaction logs in databases). In computer implementations
In delta synchronization, there are 2 main tasks
Record the deltas (or small difference of data between nodes)
Transmit these differences to the other nodes so they can process them
Implementing Delta-based synchronization for relational databases
The schema above represents a blog’s database, it’s a really simple schema so its easy to understand, now this is the scenario
We have the main database that we will call “Master” and 2 other databases named client A and client B.
Now let insert data in the master
Each DML statement should be converted in a delta (or a data difference)
Δ1
Δ2
Δ3
Copy deltas Δ 1, Δ 2, Δ 3 to the clients
So, after processing the deltas on each client, the data in all databases should look like the picture below
So that’s it for this post, in the next post we will be examing each part that is necessary to do delta-based synchronization
If you are running Xaf Blazor in ubuntu 18.04 you might have seen the following exception
The type initializer for ‘Gdip’ threw an exception.
at DevExpress.ExpressApp.Actions.ActionBase.OnHandleException(Exception e) at DevExpress.ExpressApp.Actions.ActionBase.ExecuteCore(Delegate handler, ActionBaseEventArgs eventArgs) at DevExpress.ExpressApp.Actions.PopupWindowShowAction.DoExecute(Window window) at DevExpress.ExpressApp.Actions.PopupWindowShowAction.DialogController_Accepting(Object sender, DialogControllerAcceptingEventArgs e) at DevExpress.ExpressApp.SystemModule.DialogController.Accept(SimpleActionExecuteEventArgs args) at DevExpress.ExpressApp.SystemModule.DialogController.acceptAction_OnExecute(Object sender, SimpleActionExecuteEventArgs e) at DevExpress.ExpressApp.Actions.SimpleAction.RaiseExecute(ActionBaseEventArgs eventArgs) at DevExpress.ExpressApp.Actions.ActionBase.ExecuteCore(Delegate handler, ActionBaseEventArgs eventArgs)
The error is caused by missing dependency, so the DotNet runtime itself will throw that exception. Also, I want to highlight that the exception is not related to XAF, you can read more about this problem here https://github.com/dotnet/runtime/issues/27200
To get the missing dependency just open a console and run the following commands
Here are some recommendations to host your new shiny aspnet core app on Linux in this case in Ubuntu 18.04
First, create a user with the name aspnetapp
sudo adduser myaspnetapp
after executing the command, you will have a new folder in your home directory the folder will have the same name as your username so in this case “myaspnetapp”
now let’s SSH to with the new user you just created you can do that using your favorite SSH client, for example, if you are using windows you can use putty
when you log in with the new user you will be in its home folder, now we can create a folder called app with the following command
There are many options to upload a zip file but I think is the best way is to use the secure copy command from linux “scp”, I won’t explain how you should call the scp command but if you are using windows you can run that command from the WSL console and if you are using Linux the command is already built-in, anyway here is an article about it https://linuxize.com/post/how-to-use-scp-command-to-securely-transfer-files/
Here I will write an example of how the scp command should look like and you adjust it to your needs
so that command above will do the following, it will copy the file publish.zip from the local folder to a server running on the following the IP 200.35.15.25 and the folder “/home/myaspnetapp/app”
now let’s unzip the content of the folder zip with the following command
unzip publish.zip
What we have done so far:
We have created a user in the OS
We have created a folder to host our application within the user home folder
We have uploaded a zip file containing our application the folder “/home/myaspnetapp/app”
Now that the app is already in the server we need to change the permission of the folder where the app lives to 0777, you can learn more about Linux file system permissions here https://www.guru99.com/file-permissions.html
Creating a service to monitor the app
The next step to monitor our app is to use systemd is an init system that provides many powerful features for starting, stopping, and managing processes.
Let’s start by creating a service file in the following path “/etc/systemd/system/”
here is how the content of the file should look like
[Unit]
Description=A description of your app
[Service]
WorkingDirectory=/home/myaspnetapp /app
ExecStart= /home/ myaspnetapp /app/MyExecutableFile
Restart=always
# Restart service after 10 seconds if the dotnet service crashes:
RestartSec=10
KillSignal=SIGINT
SyslogIdentifier= MyExecutableFile
User=apache
Environment=ASPNETCORE_ENVIRONMENT=Production
[Install]
WantedBy=multi-user.target
Here is a little explanation of what you might need to change if the file above
WorkingDirectory: this is the working directory, usually is the same where the app lives
ExecStart: This is the executable file how what will you write here will depend on your application if its self-contained you just need to write the name the full path of the executable file otherwise you need to write the path to the dotnet runtime and then the path to your dll as show below:
RestartSec: this is the time to wait before trying to restart the app after if the process crashes
SyslogIdentifier: the app identifier for sys logs
User: this is really important since the app will run under this user privileges, so you need to make sure that the user exists and that is able to access the files needed to start the app
That is all that we need for the service file now we need to go back to the console and enable our new service, you can do that with the following command
sudo systemctl enable MyExecutableFile.service
To start and stop the service you can use the following commands
Lately, I have been playing a lot with TestCafe which is a testing tool for website, there are 2 parts of TestCafe, the first part is TestCafe studio which is a tests editor tool where you can create or edit new tests that will be eventually used in the TestCafe test runner, both TestCafe and the test runner runs on Windows, MacOS, and Linux.
So what we are going to do today is to install TestCafe on Ubuntu Linux 18.04, in this tutorial we will only install the test runner because, in the end, my main goal is to use TestCafe to emulate the load on a server, so let’s get started
First, we need to install NodeJS and NPM, this is kind of tricky depending on your OS version, for example, if you run this command :
sudo apt -y install nodejs
in Ubunto 18.04 you will end up installing NodeJS version 8 something… that’s too old for what we need to do, so first let’s install the latest version of NodeJS, in this case, this is version 12
4) Now add the newly created mode for your display device, the structure of the command is “xrandr –addmode DisplayId Resolution”, you should replace the DisplayId for the id you got in step 1 and the resolution from what you got in
xrandr –addmode HDMI-2 “2560x1080_60.00”
Add New Resolution To Display
5) Set the new resolution
Set The New Resolution
6) So far we added the new resolution and associated it to the monitor that we want, this is not permanent it will just work until you reboot, so to make these changes permanent you need to edit your profile using the following command
gedit ~/.profile
Paste the commands from step 3 and 4 and save the file
Sometimes you want to send a zip with your visual studio solution or project, but after you archive your files you realize that the file size is huge, well worry no more you can use the following script (a .bat file) to clear all the temporary files in your visual studio solution
just save the following code in a ms-dos bat file, name the file ClearTeam.bat
rem start of the script
@echo off
for /d /r . %%d in (bin,obj,packages) do @if exist "%%d" rd /s/q "%%d"
FOR /R %%H IN (*.log) DO del "%%H"
FOR /r %%G IN (*.bak) DO del "%%G"
FOR /R %%J IN (*.suo) DO del "%%J"
IF EXIST .vs rd .vs /s/q
rem end of the script
to run it, just double click in the bat file or execute it but call the file name in console, in this case, ClearTeam.bat
I have been using XPO from DevExpress since day one. For me is the best O.R.M in the dot net world, so when I got the news that XPO was going to be free of charge I was really happy because that means I can use it in every project without adding cost for my customers.
Nowadays all my customer needs some type of mobile development, so I have decided to master the combination of XPO and Xamarin
Now there is a problem when using XPO and Xamarin and that is the network topology, database connections are no designed for WAN networks.
Let’s take MS SQL server as an example, here are the supported communication protocols
TCP/IP.
Named Pipes
To quote what Microsoft web site said about using the protocols above in a WAN network
In a fast-local area network (LAN) environment, Transmission Control Protocol/Internet Protocol (TCP/IP) Sockets and Named Pipes clients are comparable with regard to performance. However, the performance difference between the TCP/IP Sockets and Named Pipes clients becomes apparent with slower networks, such as across wide area networks (WANs) or dial-up networks. This is because of the different ways the interprocess communication (IPC) mechanisms communicate between peers.”
So, what other options do we have? Well if you are using the full DotNet framework you can use WCF.
So, it looks like WCF is the solution here since is mature and robust communication framework but there is a problem, the implementation of WCF for mono touch (Xamarin iOS) and mono droid (Xamarin Android)
You can read about Xamarin limitations in the following links
I don’t want to go into details about how the limitation of each platform affects XPO and WCF but basically the main limitation is the ability to use reflection and emit new code which is needed to generate the WCF client, also in WCF there are problems in the serialization behaviors.
So basically, what we need to do is to replace the WCF layer with some other technology to communicate to the database server
The technology I’ve selected for this AspNetCore which I would say is a really nice technology that is modern, multi-platform and easy to use. Here below you can see what is the architecture of the solution
AspNetCore
Rest API
So, what we need basically is to be able to communicate the data layer with the data store through a network architecture.
The network architecture that I have chosen is a rest API which is one of the strong fronts of AspNetCore. The rest API will work as the server that forward the communication from XPO to the Database and vice versa, you can find a project template of the server implementation here https://www.jocheojeda.com/download/560/ this implementation references one nuget where I have written the communication code, you can fine the nuget here https://nuget.bitframeworks.com/feeds/main/BIT.Xpo.AgnosticDataStore.Server/19.1.5.1