#StackBounty: #python #pandas #python-xarray Sparse DataArray Xarray search

Bounty: 50

Using DataArray objects in xarray what is the best way to find all cells that have values != 0.

For example in pandas I would do

df.loc[df.col1 > 0]

My specific example I’m trying to look at 3 dimensional brain imaging data.

first_image_xarray.shape
(140, 140, 96)
dims = ['x','y','z']

Looking at the documentation for xarray.DataArray.where it seems I want something like this:

first_image_xarray.where(first_image_xarray.y + first_image_xarray.x  > 0,drop = True)[:,0,0]

But I still get arrays with zeros.

<xarray.DataArray (x: 140)>
array([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0., -0.,  0., -0.,
        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.])
Dimensions without coordinates: x

Also – a side question – why are there some negative zeros? Are these values rounded and -0. is actually equal to something like -0.009876 or something?


Get this bounty!!!

#StackBounty: #python #r #post #recaptcha #httr Submit a form using POST with g-recaptcha-response argument

Bounty: 50

I want to submit a form from following web page: http://www.hzzo-net.hr/statos_OIB.htm

First, I use 2captcha service to bypass recaptcha:

# parameters
api_key <- "c+++"
api_url <- "http://2captcha.com/in.php"
site_key <- "6Lc3SAgUAAAAALFnYxUbXlcJ8I9grvAPC6LFTKQs"
hzzo_url <- "http://www.hzzo-net.hr/statos_OIB.htm"

# GET method
req_url <- paste0("http://2captcha.com/in.php?key=", api_key,"&method=userrecaptcha&googlekey=", 
                  site_key, "&pageurl=", hzzo_url)
get_response <- POST(req_url)
hzzo_content <- content(get_response)
hzzo_content <- xml_text(hzzo_content)
captcha_id <- stringr::str_extract_all(hzzo_content[[1]], "\d+")[[1]]

# solve captcha
Sys.sleep(16L)
captcha2_solve <- function(apiKey, capstchaID){
  req_url <- paste0("http://2captcha.com/res.php?key=", api_key,"&action=get&id=", capstchaID)
  result <- GET(req_url)
  captcha_content <- content(result)
  hzzo_response <- xml_text(captcha_content)
  hzzo_response <- strsplit(hzzo_response, "\|")
  return(hzzo_response)
  # hzzo_response <- hzzo_response[[1]][[2]]
  # return(hzzo_response)
}
hzzo_response <- captcha2_solve(api_key, captcha_id)
while(hzzo_response[[1]] == "CAPCHA_NOT_READY"){
  Sys.sleep(16L)
  hzzo_response <- captcha2_solve(api_key, captcha_id)
  return(hzzo_response)
}
hzzo_response <- hzzo_response[[1]][[2]]

After executing this code I got the response that I have enter to textarea of recaptcha. This part works fine, as I expected. The response looks like this:

"03AHqfIOmo9BlCsCKyg-lDes4oW-U3PWgCtATRUqXFcEV032acDgGoOzrV8GiZNDzCF4TbCVLcY8HZ8hR1JqO11YdRExvgPDL0EUsjCZdI0rUm_LnBRRifyb66X7V6r4n8CIm1si3EKmw36XIcZK7MGrHSNWRrj2aGzWAYO8ceobViOICOhkYe9Bsfv64tUHWvHSqNIoesD_FHplbWG3B0eMag5341NyycjpNLxgNCwVzA8mhCU3oQUcloze-mIclFMZ7J_nbVhXdy8-qipF5ZFH4xIhSQXHH-TqxyaGQFjKdgLch7MuDEQVRcQGo1o4QuSEoeCTjlPn3Mah5vC8zKrnqfbMgiOVOIDJFGvFY4KOivbBzYTz5nW9g"

After that, I should submit the form. This is the part I can’t get right.

I tried to add all arguments to POST:

parameters <- list(
  'upoib' = "93335620125", # example of number to enter
  'g-recaptcha-response' = hzzo_response
)

test <- POST(
  "http://www.hzzo-net.hr/statos_OIB.htm",
  body = toJSON(parameters), 
  encode = "json",
  verbose()
)

but this just give me the initial page.

How can I submit the form if I have the recaptcha response variable? Is it possible to submit it with httr package or I have to use Selenium. The code can be in R or Python (just need the last part, POST function).


Get this bounty!!!

#StackBounty: #python #python-3.x #distutils2 Python 3.5 create .rpm with pinstaller generated executable

Bounty: 50

I’ve got a build generated with a pyinstaller.
I need to create .rpm package which will put the executable into the /usr/bin/ and create a systemd service which will run that executable.

I found this
https://docs.python.org/3/distutils/builtdist.html and https://docs.python.org/2.0/dist/creating-rpms.html

However it doesn’t give me a full picture.

  1. Is it possible to make it?

  2. What toolset do i need to use? (Basically, how to make it).

  3. If possible – sample code


Get this bounty!!!

#StackBounty: #python #opencv #scikit-learn Method to determine polygon surface rotation from top-down camera

Bounty: 50

I have a webcam looking down on a surface which rotates about a single-axis. I’d like to be able to measure the rotation angle of the surface.

The camera position and the rotation axis of the surface are both fixed. The surface is a distinct solid color right now, but I do have the option to draw features on the surface if it would help.

Here’s an animation of the surface moving through its full range, showing the different apparent shapes:

enter image description here

My approach thus far:

  1. Record a series of “calibration” images, where the surface is at a known angle in each image
  2. Threshold each image to isolate the surface.
  3. Find the four corners with cv2.approxPolyDP(). I iterate through various epsilon values until I find one that yields exactly 4 points.
  4. Order the points consistently (top-left, top-right, bottom-right, bottom-left)
  5. Compute the angles between each points with atan2.
  6. Use the angles to fit a sklearn linear_model.linearRegression()

This approach is getting me predictions within about 10% of actual with only 3 training images (covering full positive, full negative, and middle position). I’m pretty new to both opencv and sklearn; is there anything I should consider doing differently to improve the accuracy of my predictions? (Probably increasing the number of training images is a big one??)

I did experiment with cv2.moments directly as my model features, and then some values derived from the moments, but these did not perform as well as the angles. I also tried using a RidgeCV model, but it seemed to perform about the same as the linear model.


Get this bounty!!!

#StackBounty: #machine-learning #python #neural-network #classification #supervised-learning Recognising made up terms

Bounty: 50

Say I have a tagging system on electrical circuit:

Name Description

BT014Battery. Power source

Name          Description
BT104         Battery. Power source
SW104         Circuit switch
LBLB-F104     Fluorescent light bulb
LBLB104       Light bulb
...           ...

I have a hundreds of tags created by people who should have followed my naming conventions but sometimes they make mistakes add unnecessary additional characters to tag names (i.e. BTwq104 etc.).

Up until now I used regular expressions, that I built over time whilst observing various inconsistencies that users introduce whilst naming different parts of their curuits, to parse the names and tell me what the different elements are. For example: name ‘BT104‘ would tell me its a battery on circuit 104.

I would like to investigate or use a machine learning technique to identify what a tag name is (same way I used reg ex’s). Any suggestions and approaches are welcome.

So far I tried Named-entity recognition suggested technique “Bag of words“. Followed a few tutorials here and here (latter being the most useful in learning). None of them produced wanted results if any. I think that “Bag of words” are mostly used for real word rather than made up words.

Thanks


Get this bounty!!!

#StackBounty: #python #matplotlib #plot #3d Covering 2D plots with 3D surface in python

Bounty: 50

I am attempting to take many xz plots, each at various y values, and overlay a surface. I have seen many examples of how to plot 3D surfaces in python, but aside from this post, nothing seems to match my inquiry very closely.

The image of what I need to do is shown below (Note: ignore the “constant x” – this is due to a more complicated variable layout than I explained here):

Magnitude vs. Frequency

My code is as follows, and simply takes data and plots each of the individual Magnitude vs. Frequency plots (xz plots):

import numpy as np
import glob, os
import codecs
import re
import matplotlib.pyplot as plt
#-------------------------------------------------------------------------                   
os.chdir('C:/Users/elarrick/Desktop/201807161056')
dataFolders = glob.glob('./*.ltda')

dataLines = []
freq = []
OpenLoopMag = []
OpenLoopPhase = []
for item in dataFolders:
    name, ext = os.path.splitext(item)
    if ext == '.ltda':
        print item
        dataLines = []
        f = codecs.open(item, encoding='utf-8')
        for line in f:
            if '<p>' in line:
                dataLines.append(line)      #All lines with <p> are an entry in dataLines
        #print "nn", dataLines
        #break
        for item in dataLines:
            item = re.sub("<p>", "", item)
            item = re.sub("True</p>", "", item)
            item = item.replace(",", "")
            splitItem = item.split()
            #print splitItem
            freq.append(float(splitItem[0]))
            OpenLoopMag.append(float(splitItem[1]))
            OpenLoopPhase.append(float(splitItem[2]))
        print "Frequencies: ", freq
        print "nnnnnnOpenLoopMag: ", OpenLoopMag
#   This is where I will make the plots for each x,y position
        name = name.strip(".\")
        name = name.replace("-","NEG")
        plt.semilogx(freq, OpenLoopMag)
        #plt.plot(freq, OpenLoopMag)
        plt.xlabel("Frequency, (Hz)")
        plt.ylabel("Magnitude")
        plt.title("{0}".format(name))
        plt.xlim([20,2000])
        #plt.ylim([-43.2,10.9])
        ticks = [20,40,70,100,200,400,700,1000,2000]
        plt.xticks(ticks,ticks)
        plt.savefig("plot_{0}.png".format(name))

#________ Clear the values for the next data folder_______#            
        freq = []
        OpenLoopMag = []
        OpenLoopPhase = []
        break
    else:
        print "Something went wrong - check ColorMap.py"
        sys.exit()

The next thing I need is to grab each plot, find the y value at which the data was taken, and plot along a y axis (shown coming out of the page in the previous picture). Can you help me do that?


Get this bounty!!!

#StackBounty: #python #django #django-rest-framework Django Rest Framework – passing Model data through a function, then posting output…

Bounty: 50

(Django 2.0, Python 3.6, Django Rest Framework 3.8)

I’m trying to fill the calendarydays field in the model below:

Model

class Bookings(models.Model):
    booked_trainer = models.ForeignKey(TrainerProfile, on_delete=models.CASCADE)
    booked_client = models.ForeignKey(ClientProfile, on_delete=models.CASCADE)
    trainer_availability_only = models.ForeignKey(Availability, on_delete=models.CASCADE)
    calendarydays = models.CharField(max_length=300, blank=True, null=True)

    PENDING = 'PENDING'
    CONFIRMED = 'CONFIRMED'
    CANCELED = 'CANCELED'

    STATUS_CHOICES = (
        (PENDING, 'Pending'),
        (CONFIRMED, 'Confirmed'),
        (CANCELED, 'Canceled')
    )


    booked_status = models.CharField(
        max_length = 9,
        choices = STATUS_CHOICES,
        default = 'Pending'
    )

    def __str__(self):
        return str(self.trainer_availability_only)

Now, I have a function that takes values from trainer_availability_only and converts those values to a list of datetime strings, the returned output would look like this:

{‘calendarydays’: [‘2018-07-23 01:00:00’, ‘2018-07-23 02:00:00’,
‘2018-07-23 03:00:00’, ‘2018-07-30 01:00:00’, ‘2018-07-30 02:00:00’,
‘2018-07-30 03:00:00’, ‘2018-08-06 01:00:00’, ‘2018-08-06 02:00:00’,
‘2018-08-06 03:00:00’, ‘2018-08-13 01:00:00’, ‘2018-08-13 02:00:00’,
‘2018-08-13 03:00:00’, ‘2018-08-20 01:00:00’, ‘2018-08-20 02:00:00’,
‘2018-08-20 03:00:00’]}

Problem

How can I fill the calendarydays field with the function output for a user to select from a dropdown, and where should I implement this logic (in my view or the serializer)? My main point of confusion is that, because my function depends on data from trainer_availability_only, I don’t want to create a separate model/table for this information (as that would seem too repetitive). I also don’t fully understand where in my serializers or views I can implement some sort of dropdown for a User to choose a single calendarydays value for (like I would be able to for a ForeignKey or OneToOneField for example).

Details for the other models aren’t really relevant to the question, except trainer_availability_only, which basically gives the user a dropdown selection that would look like this:

('Monday','12:00 am - 1:00 am')
('Wednesday','4:00 pm - 5:00 pm')
etc.

Any help is greatly appreciated.


Get this bounty!!!

#StackBounty: #linux #ubuntu #permissions #python 'std::bad_alloc' after mistakenly changing /usr/ permissions

Bounty: 50

I’m working on a Linux machine running Ubuntu Bionic Beaver, release 18.04.

The other day I mistakenly changed the /usr/ directory to be owned by a user, instead of root. Unfortunately, I did that recursively, and so messed quite a bit of the system up because it also changed the suid permissions on some of the commands (e.g. passwd, sudo). We really can’t reinstall (well we can but it’ll cost!), so I booted from a LiveUSB, and changed manually all the correct user/group/permissions for each file that I could identify had a non-Root:Root User:Group. I did this by comparing the output of another Ubuntu computer of ls -lha /usr/.

It seems to be mostly fixed, but now I’m running into the error ‘std::bad_alloc’ after running some pretty standard python scripts. The strange part about this is that it only comes up sometimes. For example, if I open python from the command line and copy and paste code, the code will all run fine with no error. However if I run the entire script from the command line (e.g. python script.py) then I get this error. The full error message is:

terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
Aborted (core dumped)

But to add another twist – sometimes I can run the same python script from the command line with no problem, and others I get this error as above.

If anybody has ideas as to where to specifically look to fix this that’d be great! I’m going to try and do the same thing as before but with the ls -lha /usr/ output from an 18.04 release, as I only had a 16 release output on hand.


Get this bounty!!!

#StackBounty: #linux #ubuntu #permissions #python 'std::bad_alloc' after mistakenly changing /usr/ permissions

Bounty: 50

I’m working on a Linux machine running Ubuntu Bionic Beaver, release 18.04.

The other day I mistakenly changed the /usr/ directory to be owned by a user, instead of root. Unfortunately, I did that recursively, and so messed quite a bit of the system up because it also changed the suid permissions on some of the commands (e.g. passwd, sudo). We really can’t reinstall (well we can but it’ll cost!), so I booted from a LiveUSB, and changed manually all the correct user/group/permissions for each file that I could identify had a non-Root:Root User:Group. I did this by comparing the output of another Ubuntu computer of ls -lha /usr/.

It seems to be mostly fixed, but now I’m running into the error ‘std::bad_alloc’ after running some pretty standard python scripts. The strange part about this is that it only comes up sometimes. For example, if I open python from the command line and copy and paste code, the code will all run fine with no error. However if I run the entire script from the command line (e.g. python script.py) then I get this error. The full error message is:

terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
Aborted (core dumped)

But to add another twist – sometimes I can run the same python script from the command line with no problem, and others I get this error as above.

If anybody has ideas as to where to specifically look to fix this that’d be great! I’m going to try and do the same thing as before but with the ls -lha /usr/ output from an 18.04 release, as I only had a 16 release output on hand.


Get this bounty!!!

#StackBounty: #linux #ubuntu #permissions #python 'std::bad_alloc' after mistakenly changing /usr/ permissions

Bounty: 50

I’m working on a Linux machine running Ubuntu Bionic Beaver, release 18.04.

The other day I mistakenly changed the /usr/ directory to be owned by a user, instead of root. Unfortunately, I did that recursively, and so messed quite a bit of the system up because it also changed the suid permissions on some of the commands (e.g. passwd, sudo). We really can’t reinstall (well we can but it’ll cost!), so I booted from a LiveUSB, and changed manually all the correct user/group/permissions for each file that I could identify had a non-Root:Root User:Group. I did this by comparing the output of another Ubuntu computer of ls -lha /usr/.

It seems to be mostly fixed, but now I’m running into the error ‘std::bad_alloc’ after running some pretty standard python scripts. The strange part about this is that it only comes up sometimes. For example, if I open python from the command line and copy and paste code, the code will all run fine with no error. However if I run the entire script from the command line (e.g. python script.py) then I get this error. The full error message is:

terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
Aborted (core dumped)

But to add another twist – sometimes I can run the same python script from the command line with no problem, and others I get this error as above.

If anybody has ideas as to where to specifically look to fix this that’d be great! I’m going to try and do the same thing as before but with the ls -lha /usr/ output from an 18.04 release, as I only had a 16 release output on hand.


Get this bounty!!!