#StackBounty: #c# #ado.net #bigdata #datareader What is the best way to load huge result set in memory?

Bounty: 50

I am trying to load 2 huge resultsets(source and target) coming from different RDBMS but the problem with which i am struggling is getting those 2 huge result set in memory.

Considering below are the queries to pull data from source and target :

  Sql Server -  select Id as LinkedColumn,CompareColumn from Source order by LinkedColumn

  Oracle -select Id as LinkedColumn,CompareColumn from Target order by LinkedColumn

Records in Source : 12377200

Records in Target : 12266800

Following are the approaches i have tried with some statistics:

1) 2 open data reader approach for reading source and target data:

Total jobs running in parallel = 3

Time taken by Job1 = 01:47:25

Time taken by Job1 = 01:47:25

Time taken by Job1 = 01:48:32

There is no index on Id Column.

Major time is spend here :


var dr = command.ExecuteReader();

Problems : 
There are timeout issues also for which i have to kept `commandtimeout` to 

0(infinity) and it is bad.

2) Chunk by chunk reading approach for reading source and target data:

   Total jobs = 1
   Chunk size : 100000
   Time Taken : 02:02:48
   There is no index on Id Column.

3) Chunk by chunk reading approach for reading source and target data:

   Total jobs = 1
   Chunk size : 100000
   Time Taken : 00:39:40
   Index is present on Id column.

4) 2 open data reader approach for reading source and target data:

   Total jobs = 1
   Index : Yes
   Time: 00:01:43

5) 2 open data reader approach for reading source and target data:

   Total jobs running in parallel = 3
   Index : Yes
   Time: 00:25:12

I does observe that while having index on LinkedColumn does improve performance but the problem is we are dealing with 3rd party RDBMS tables which might have index or might not.

We would like to keep database server as free as possible so data reader approach doesnt seems good idea because there will be lots of jobs running in parallel which will put so much pressure on database server which we dont want.

Hence we want to fetch records in my resource memory from source to target and do 1 – 1 records comparision keeping database server free.

Note : I want to do this in my c# application and dont want to use SSIS or Linked Server.

Update :

Source Sql Query Execution time in sql server management studio: 00:01:41

Target Sql Query Execution time in sql server management studio:00:01:40

What will be the best way to read huge result set in memory ?

Code :

static void Main(string[] args)
        {   
            // Running 3 jobs in parallel
             //Task<string>[] taskArray = { Task<string>.Factory.StartNew(() => Compare()),
        //Task<string>.Factory.StartNew(() => Compare()),
        //Task<string>.Factory.StartNew(() => Compare())
        //};
            Compare();//Run single job
            Console.ReadKey();
        }
public static string Compare()
        {
            Stopwatch stopwatch = new Stopwatch();
            stopwatch.Start();
            var srcConnection = new SqlConnection("Source Connection String");
            srcConnection.Open();
            var command1 = new SqlCommand("select Id as LinkedColumn,CompareColumn from Source order by LinkedColumn", srcConnection);
            var tgtConnection = new SqlConnection("Target Connection String");
            tgtConnection.Open();
            var command2 = new SqlCommand("select Id as LinkedColumn,CompareColumn from Target order by LinkedColumn", tgtConnection);
            var drA = GetReader(command1);
            var drB = GetReader(command2);
            stopwatch.Stop();
            string a = stopwatch.Elapsed.ToString(@"d.hh:mm:ss");
            Console.WriteLine(a);
            return a;
        }
      private static IDataReader GetReader(SqlCommand command)
        {
            command.CommandTimeout = 0;
            return command.ExecuteReader();//Culprit
        }


Get this bounty!!!

#StackBounty: #c# #asp.net-web-api #asp.net-web-api2 #odata Disabling OData V4 meta data and controllers at runtime

Bounty: 50

We have several modules in our software that ship as a single product. When a module is activated, those features become available. We would like our OData APIs to follow the same pattern. However I can’t figure out how to make the $metadata ignore controllers for modules that have been disabled. Basically I want to determine what is available at any time instead of application start time.

We are using the following type of cod to register the routes:

    static public void Register(HttpConfiguration config)
    {
        config.MapHttpAttributeRoutes();
        var builder = new ODataConventionModelBuilder();
        builder.EntitySet<Module1Entity>("Module1Entities");
        builder.EntitySet<Module2Entity>("Module2Entities");
        config.MapODataServiceRoute("odata", "api", builder.GetEdmModel());
    }

    protected void Application_Start(object sender, EventArgs e)
    {
        GlobalConfiguration.Configure(Register);
    }

So we only want Module1Entity to show up in the metadata if the module has been activated. We already have code to disable the associated controller when the module is deactivated.

Any ideas?


Get this bounty!!!

#StackBounty: #c# #asp.net #iis #iis-8 Unable to get the current user identity value

Bounty: 50

I have a intranet website where I am able to get the user’s identity value when run in Local machine.

System.Web.HttpContext.Current.User.Identity.Name.ToString().Substring(3)

But When I deploy the same on IIS 8.5, It finds it blank.

Please help me understand where I am going wrong ?

For impersonation we have to use the specific username and password.

Web.config:

<system.web>
    <authentication mode="Windows" />
        <customErrors mode="RemoteOnly" defaultRedirect="~/Pages/Error/Error.aspx" >
            <error statusCode="404" redirect="~/Pages/Error/404.aspx" />
        </customErrors>
    <identity impersonate="true" userName="user1" password="pass1" />
  </system.web>
<system.webServer>
    <modules runAllManagedModulesForAllRequests="true" />
    <validation validateIntegratedModeConfiguration="false" />
</system.webServer>

IIS settings:

Windows authentication - Enabled
Impersonation  - Enabled
Rest all disabled.

Default app pool - Integrated mode.


Get this bounty!!!

#StackBounty: #c# #ios #xamarin #xamarin.forms Customizing a Progress Bar appearance in Xamarin.Forms

Bounty: 50

I used Drawable to customize the rendering of the ProgressBar in Android as answered to this question but the solution is not working with iOS.

Below is how it renders in Android.
enter image description here

Below is how it renders in iOS
enter image description here

Below is the code for my iOS CustomRenderer

[assembly: ExportRenderer(typeof(CustomProgressbar), typeof(CustomProgressBarRenderer))]
namespace Demo.iOS.Renderers
{
public class CustomProgressBarRenderer : ProgressBarRenderer
{
    protected override void OnElementChanged(ElementChangedEventArgs<ProgressBar> e)
    {
        try
        {
            base.OnElementChanged(e);

            if (Control != null)
            {                
               Control.ProgressTintColor = Color.FromHex("#ff0000").ToUIColor();                       
               Control.TrackTintColor = Color.FromHex("#3489cc").ToUIColor();
             } 
        }
        catch (Exception ex)
        {

        }
    }

    public override void LayoutSubviews()
    {
        base.LayoutSubviews();
        var X = 1.0f;
        var Y = 15.0f;
        CGAffineTransform _transform = CGAffineTransform.MakeScale(X, Y);
        this.Transform = _transform;
        this.ClipsToBounds = true;
        this.Layer.MasksToBounds = true;
        this.Layer.CornerRadius = 5;
    }
}

}

How do I accomplish this?


Get this bounty!!!

#StackBounty: #c# #asp.net #wordpress have few pages on a PHP site powered by ASP.net

Bounty: 50

I have a basic wordpress site. What i want is for the user to click on Add to Cart button and be taken to the shopping cart flow which is built in ASP.net.

Few Facts:

  1. WordPress site is on server #1 (linux)
  2. ASP.net site is on server#2 (windows)

Minimum Features:

  1. Be able to keep the same domain name (ie: “www.example.com/” should be the same for both the sites.
  2. Possibly some way to share cookies

What i have researched:

  1. Using iframes plugin in wordpress. Not sure if want to do this. I would prefer to keep the pages totally seperate. – https://wordpress.org/plugins/iframe-less-plugin/

What is the process of having one site use pages from two different servers and the user not know that they are going from one server to another?


Get this bounty!!!

#StackBounty: #unity #c# #collision-detection How to efficiently keep the player from moving outside the gameobject he is colliding with?

Bounty: 50

I am developing a game with Unity 2017.
The game mechanics are as follow:

  • Classic top-down RPG;
  • Movements are: UP, DOWN, LEFT and RIGHT;
  • Player evolves in a 16*16 map;
  • He can dig in the direction he wants to make his way through the map one tile at a time;

At first I was using a very classical approach using tilemaps, RigidBody2D and BoxCollider2D on tiles and the player GameObjects.

Collision detections went fine and my tilemap looked pretty much like that (red being the walls and white the empty spaces):

Classic tilemap

The problem with this approach is that I have to keep in memory the state of 256 tiles in a List<Vector2f> and to create the 256 expected GameObjects representing either my walls (which a totally black), or my floor (the empty space).

Creating a black GameObject with its own BoxCollider2D for each of the tile seems a little bit overkill. And this brings me to the second iteration of my game: to keep in memory only the tiles the player digged.

Expected tilemap

What I would like to achieve efficiently, is to let the player evolve as long as he is colliding with an underlying tile, but stop moving as soon as he doesn’t.

This is ok:

This is ok

This isn’t, and player should be placed to previous state.

This isn't

I could take advantage of OnTriggerEnter2d and handle custom collision and response, but I have the feeling that something more simple and more optimized may exists somehwhere in the wild!

Thanks to any help provided!


Get this bounty!!!

#StackBounty: #c# #.net #ssl Which TLS version was negotiated?

Bounty: 100

I have my app running with .NET 4.7. By default, it will try to use TLS1.2.
However, is it possible to know in this code which TLS version was negotiated when performing, for example, an HTTP Request as below ?

HttpWebRequest request =
                (HttpWebRequest)WebRequest.Create(decodedUri);

    if (requestPayload.Length > 0)
            {
                using (Stream requestStream = request.GetRequestStream())
                {
                    requestStream.Write(
                        requestPayload,
                        0,
                        requestPayload.Length);
                }
            }


Get this bounty!!!

#StackBounty: #c# #wcf #coldfusion What role does a public certificate play in WCF?

Bounty: 50

We have our own certificate that we use as part of the ClientCredentials in Transport Client Credentials as seen below.

WSHttpBinding wsBinding = new WSHttpBinding();
wsBinding.Security.Mode = System.ServiceModel.SecurityMode.Transport;
wsBinding.Security.Transport.ClientCredentialType = HttpClientCredentialType.Certificate;
wsClient = new WSService.WSClient(wsBinding, new EndpointAddress(serviceURL));
wsClient.ClientCredentials.ClientCertificate.SetCertificate(StoreLocation.LocalMachine, StoreName.My, X509FindType.FindBySubjectName, clientCertificateSubjectName);

Our partner also provided us with a copy of their certificate that is used on the server in order to validate their server. Calls to the service currently succeed without us having installed or done anything with this certificate.

We are using ColdFusion and generally we have to install these into the java certificate store – but its odd to me that service communication is working under Visual Studio when I have not done anything with the service’s certificate

What is the role/purpose of the service certificate? Do clients install it in the MMC Certificate Trusted People in or reference it in the client-side configuration e.g. WSHttpBinding above?


Get this bounty!!!

#StackBounty: #c# #unit-testing #reactiveui Unit Testing ItemChanged on ReactiveList Triggering Property Change

Bounty: 100

Boiled down: I need to delay execution in my unit test thread so that an observable has time to update a property. Is there a Reactive way to do this without having to resort to Thread.Sleep?

I have a ViewModel that manages a ReactiveList of “Thing”. Both ThingViewModel and the main ViewModel are derived from ReactiveObject. Thing has an IsSelected property and the ViewModel has a SelectedThing property that I keep synched based on observing changes on IsSelected. I have a couple of different views that use the ViewModel and this allows me to synch the selected Things between those views nicely. And it works. The problem comes when I try to unit test this interaction.

The ViewModel has this subscription in its constructor:

Things.ItemChanged.Where(c => c.PropertyName.Equals(nameof(ThingViewModel.IsSelected))).ObserveOn(ThingScheduler).Subscribe(c =>
{
    SelectedThing = Things.FirstOrDefault(thing => thing.IsSelected);
});

In my unit test, this assert always fails:

thingVM.IsSelected = true;
Assert.AreEqual(vm.SelectedThing, thingVM);

But this assert always passes:

thingVM.IsSelected = true;
Thread.Sleep(5000);
Assert.AreEqual(vm.SelectedThing, thingVM);

Essentially, I need to wait long enough for the subscription to complete the change. (I don’t really need that long to wait as when running in the UI, it’s pretty snappy.)

I tried adding an observer in my unit test to wait on that processing, hoping they’d be similar. But that’s been a wash. Here’s what didn’t work.

var itemchanged = vm.Things.ItemChanged.Where(x => x.PropertyName.Equals("IsSelected")).AsObservable();
. . . 
thingVM.IsSelected = true;
var changed = itemChanged.Next();
Assert.AreEqual(vm.SelectedThing, thingVM);

Moving the itemChanged.Next around didn’t help, nor did triggering iteration by calling .First() or .Any() on changed (both of those hung the process as it blocked the thread for a notification that never occurred).

So. Is there a Reactive way to wait on that interaction so that I can Assert that the property change is happening correctly? I messed around some with TestScheduler (I needed manual Scheduler setting for the UI anyway, so this wasn’t hard), but that doesn’t seem to apply, as the actual action given to the scheduler happens in my ViewModel on ItemChanged and I couldn’t find a way to make that trigger in the ways TestScheduler seems set up to work.


Get this bounty!!!

#StackBounty: #ffmpeg #raspberry-pi #c++ #build #ip-camera crash while building with ffmpeg and contrib in raspberry pi 3 model B(debian)

Bounty: 100

  • OpenCV => 3.3.0
    • Operating System / Platform => raspberry pi 3 model B(debian)
    • Compiler => UNIX Makefiles
    • Cmake-GUI => 3.6.2

https://pastebin.com/5VGHA92m (build output from terminal)

[ 39%] Linking CXX shared library ../../lib/libopencv_videoio.so
/usr/bin/ld: /usr/local/lib/libx264.a(common.o): relocation
R_ARM_MOVW_ABS_NC against `a local symbol’ can not be used when making
a shared object; recompile with -fPIC /usr/local/lib/libx264.a: error
adding symbols: Bad value collect2: error: ld returned 1 exit status
modules/videoio/CMakeFiles/opencv_videoio.dir/build.make:228: recipe
for target ‘lib/libopencv_videoio.so.3.3.0’ failed make[2]: *
[lib/libopencv_videoio.so.3.3.0] Error 1 CMakeFiles/Makefile2:4311:
recipe for target ‘modules/videoio/CMakeFiles/opencv_videoio.dir/all’
failed make[1]: *

[modules/videoio/CMakeFiles/opencv_videoio.dir/all] Error 2
Makefile:160: recipe for target ‘all’ failed make: *** [all] Error 2

using opencv=3.3.0
opencv_contrib=3.3.0

before building I Unchecked:
BUILD_opencv_world,
BUILD_TBB,
WITH_TBB,
BUILD_opencv_python2,
BUILD_opencv_python3,
WITH_MATLAB,
WITH_CUDA,
BUILD_opencv_dnn,
WITH_OPENGL,
WITH_QT

and Checked:
WITH_FFMPEG(Default)

I installed all libraries (mentioned in below link)
https://pastebin.com/vFkYzQ08 (prerequisite libraries before building)

I also tried remove all those libraries and building opencv source with ffmpeg and contrib (yet build crashes at 23% for opencv 3.4 and 39% for opencv 3.3)


Get this bounty!!!