#StackBounty: #performance #date #elasticsearch #filter Do additional date range filters increase the performance?

Bounty: 50

To an existing, large ElasticSearch 5 index, I want to add a date field, containing the date of the indexation of each document. Afterwards I want to query this index, to return all documents, created in the last minute.

In the ElasticSearch Ultimative Guide for version 1 it is mentioned, that adding additional filters for day, month and/or year can improve the performance drastically. Newer versions of the guide do not say so anymore.

Can I gain performance in ElasticSearch 5 with adding additional date filters?


Get this bounty!!!

#StackBounty: #performance #numerical-methods #symbolic-math #numerical-computing Symbolic vs Numeric Math – Performance

Bounty: 50

Do symbolic math calculations for solving nonlinear polynomial systems cause huge performance(calculation speed) disadvantage compared to numeric calculations? Are there any benchmark/data about this?

Found a related question: https://scicomp.stackexchange.com/questions/21754/symbolic-computation-vs-numerical-computation


Get this bounty!!!

#StackBounty: #amazon-ec2 #performance #mongodb MongoDB stalling at 100 writes/s but benchmark says 54000 writes/s

Bounty: 50

I am running MongoDB (v3.4 Community Edition) on an AWS EC2 m4.large instance. I installed it according to this MongoDB tutorial. I haven’t modified any MongoDB config. I haven’t configured any replica set or shard. I have a Jersey API which interacts with MongoDB using org.mongodb.morphia (v1.3.2) Java Driver.

I have created a load test using SoapUI where I make 100 calls to API which in turn create 100 write operations (one document is created in one collection per write operation) for MongoDB. I run the test for 24 hours. At the end, I see my webserver trying to communicate with the Mongo server but the connection is timing out and the host CPU is running at 100%. I conclude that the MongoDB server is choking.

I then tried this benchmarking exercise, still with a database hosted on an m4.large. 50,000,000 records were created in 923.133 seconds i.e. 54,113 insertions per second. That is over 500 times faster!

If MongoDB is performing so well then why it is choking at 100 insertion per second when going through JAVA driver? Is the Java driver slow? Is my use of the Java driver wrong? Is my EC2 instance size is too low? Would it help to add replication (RAID and Replica sets)?

I am new to MongoDB hosting and really appreciate your help learning.


Get this bounty!!!

#StackBounty: #amazon-ec2 #performance #mongodb MongoDB Community edition chocking at 100 write operations per second

Bounty: 50

I have installed MondoDB (v3.4) on AWS EC2 instance (m4.large) using following link. I havent modify any MongoDB config. I havent configured any replica set or shard. I have a jersey API which interact with MongoDB using JAVA driver. I created a load test using SoapUI where I made the 100 calls to API which in turn created 100 write operations (one document is created in one collection per write operation) for MongoDB. I ran the test for 24 Hours. I saw at the end my webserver was trying to communicate with Mongo Sever but connection was timing out and Host CPU was running at 100%.That is why I concluded MongoDB server is chocking.

Is my EC2 instance size is too low?
Am I have to do replication (RAID and Replica sets)?

I am going to try benchmarking exercise mentioned here. I will updated the post after. I am new to MongoDB hosting and really appreciate your help learning.

Update 1: I did the MongoDB benchmarking test above. Database hosted on M4.Large instance gave following data. 50,000,000 records were created in 923.133 seconds i.e. 54,113 insertion per second. If MongoDB is performing so well then why it is chocking at 100 insertion per second when done through JAVA driver. Is Java driver slow? Is my implementation of JAVA Driver wrong? I am using org.mongodb.morphia (v1.3.2) Java Driver.


Get this bounty!!!

#StackBounty: #mysql #performance #performance-tuning #explain A MySQL EXPLAIN number of rows discrepancy

Bounty: 50

MySQL 5.5.49-log

More questions on the query in Why does it use temporary? (MySQL) (the query is the same but the question is different):

I have the following table (filled with many rows):

CREATE TABLE `SectorGraphs2` (
  `Kind` tinyint(3) UNSIGNED NOT NULL COMMENT '1 - продюсер, 2 - жанр, 3 - регион',
  `Criterion` tinyint(3) UNSIGNED NOT NULL,
  `Period` tinyint(3) UNSIGNED NOT NULL,
  `PeriodStart` date NOT NULL,
  `SectorID` int(10) UNSIGNED NOT NULL,
  `Value` float NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_bin;

ALTER TABLE `SectorGraphs2`
  ADD UNIQUE KEY `Producer2` (`Kind`,`Criterion`,`Period`,`PeriodStart`,`SectorID`) USING BTREE,
  ADD KEY `SectorID` (`SectorID`);

then I run:

EXPLAIN 
    SELECT SectorID, SUM(Value)
    FROM SectorGraphs2
    WHERE Kind = 1 AND Criterion = 7
      AND Period = 1
      AND PeriodStart >= ? AND PeriodStart < ? + INTERVAL 1 WEEK
    GROUP BY SectorID

and it produces:

+---+---+---+---+---+---+---+---+---+---+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+---+---+---+---+---+---+---+---+---+---+
| 1 | SIMPLE | SectorGraphs2 | range | Producer2,SectorID | Producer2 | 6 | NULL | 1 | Using index condition; Using temporary; Using filesort |
+---+---+---+---+---+---+---+---+---+---+

See nicely formatted explain here.

My question: Why it is used a temporary table and filesort but it reports only 1 row examined? It seems that because of using a temporary table, it should process more than one row. How can I determine the real number of rows processed? How to solve this discrepancy about number of processed rows?

Note that the task I was assigned to do now is to eliminate heavy (involving too many rows) queries. And now I do not know how to do this.


Get this bounty!!!

#StackBounty: #machine-learning #cross-validation #performance LFW face pair-matching performance evaluation, why retrain model on view2?

Bounty: 50

I am trying to understand how performance evaluation works in LFW(Labeled Faces in the Wild) dataset http://vis-www.cs.umass.edu/lfw/.

I am interested in task: pair-matching. However, as I dig deeper, I found myself confused.

Here is a brief summary on evaluating pair-matching performance in LFW dataset:

  1. LFW dataset is divided into View1 and View2. View1 is for development of algorithms, you can use it to select model, tune parameters and choose features. View2 is for reporting accuracy of your model produced by View1.

  2. View1 description:

    For development purposes, we recommend using the below training/testing split, which was generated randomly and independently of the splits for 10-fold cross validation, to avoid unfairly overfitting to the sets above during development. For instance, these sets may be viewed as a model selection set and a validation set. See the tech report below for more details.

    pairsDevTrain.txt, pairsDevTest.txt

  3. View2 description:

    As a benchmark for comparison, we suggest reporting performance as 10-fold cross validation using splits we have randomly generated.

I also found an example of carrying out the experiment with PCA for face pair-matching in the LFW 2008 paper.

Eigenfaces for pair matching. We computed eigenvectors from the training set of View 1 and determined the threshold value for classifying pairs as matched or mismatched that gave the best performance on the test set of View 1. For each run of View 2, the training set was used to compute the eigenvectors, and pairs were classified using the threshold on Euclidian distance from View 1.

State of the art pair matching. To determine the current best performance on pair matching, we ran an implementation of the current state of the art recognition system of Nowak and Jurie [14].11 The Nowak algorithm gives a similarity score to each pair, and View 1 was used to determine the threshold value for classifying pairs as matched or mismatched. For each of the 10 folds of View 2 of the database, we trained on 9 of the sets and computed similarity measures for the held out test set, and classified pairs using the threshold

My questions are:

  1. How to do training with View1 data using 10-fold cross validation?

    The data is already split into pairsDevTrain.txt and pairsDevTest.txt. Does it mean that I need to merge these two file and then do a standard 10-fold cross validation to train my model?

  2. Why is 10-fold cross validation required in View2?

    Since model and parameter is all determined using data in View1, why not just use all View2 data to report performance.

  3. Since 10-fold cross validation is required in View2, there must be a training process. Why retrain another model?

    It is worth mentioning here, both in View1 and View2. train and test data don’t share common identity, i.e. person1 appear in train, will not appear in test.

  4. 10-fold cross validation is recommended for both View1 and View2. 10-fold splits are given for View2 but not View1. Is there a reason why?

Thank you beforehand for helping me understand the performance evaluation for LFW.


Get this bounty!!!

#StackBounty: #windows #ssd #performance #benchmarking #nvme SSD with write cache buffer flushing turned on is way slower in AS SSD but…

Bounty: 50

When running a benchmark using the program AS SSD on an NVMe drive, When the second checkbox seen in the image below is unchecked, the drive gets terrible write performance in AS SSD (see first benchmark screenshot below), but not in CrystalDiskMark (see last screenshot); however, if I check that box, then AS SSD performs well. Does anyone know what’s going on here? My concern is that according to that checkbox description, I should NOT have it checked since my drive doesn’t have its own power supply, but AS SSD is so slow, I’m concerned other programs may be affected.

In case it’s helpful, I learned about that checkbox in the first place from reading https://www.reddit.com/r/Dell/comments/628odr/toshiba_nvme_slow_write_speed_fix/, but this doesn’t explain to me why CDM would still be fast. See also Slow SSD performance (Toshiba 1GB NVMe). I updated my Intel RST drivers and that didn’t help.

I understand there could be a roundabout driver solution here, but I want to stay within warranty and try to find out what’s going on so I can get Dell to support this and issue an official solution.

enter image description here
enter image description here
enter image description here
enter image description here


Get this bounty!!!

#StackBounty: #menus #navigation #performance #cache Nav and logo loading each time causing menu to move JointsWP – Foundation 6

Bounty: 50

Hi I was wondering if anyone could help.

I’m creating a site using JointsWP Foundation 6 theme and have created a new fixed side menu which includes the logo and social links. My problem is everytime a user click on the menu it reloads causing a shift – is there a way of stopping this – is it a page load issue or have i come about it the wrong way? I tried adding a caching plugin but it hasn’t seemed to help. Any suggestions appreciated.

Here is examples of my code:

<body <?php body_class(); ?>>

    

and the page.php

<?php get_header(); ?>
    
--> <!--
--> --> <main id="main" class="large-9 medium-9 columns contentSection" role="main"> <?php if (have_posts()) : while (have_posts()) : the_post(); ?> <?php get_template_part( 'parts/loop', 'page' ); ?> <?php endwhile; endif; ?> <!--</main> <!-- end #main --> <!--</div> <!-- end #inner-content --> <!--</div> <!-- end #content -->

Edit:

i have added 2 test pages so that you can see – biggreenspace.com/test-page-1 and you will be able to navigate to test page 2 (the other menu will take you to the maintenance screen). This primarily happens in Chrome and Firefox – not in IE edge.


Get this bounty!!!

#StackBounty: #python #performance #regex #natural-language-proc #cython Using lots of regex substitutions to tokenize text

Bounty: 50

I authored a piece of code that was merged into the nltk codebase. It is full of regex substitutions:

import re
from six import text_type

from nltk.tokenize.api import TokenizerI

class ToktokTokenizer(TokenizerI):
    """
    This is a Python port of the tok-tok.pl from
    https://github.com/jonsafari/tok-tok/blob/master/tok-tok.pl

    >>> toktok = ToktokTokenizer()
    >>> text = u'Is 9.5 or 525,600 my favorite number?'
    >>> print (toktok.tokenize(text, return_str=True))
    Is 9.5 or 525,600 my favorite number ?
    >>> text = u'The https://github.com/jonsafari/tok-tok/blob/master/tok-tok.pl is a website with/and/or slashes and sort of weird : things'
    >>> print (toktok.tokenize(text, return_str=True))
    The https://github.com/jonsafari/tok-tok/blob/master/tok-tok.pl is a website with/and/or slashes and sort of weird : things
    >>> text = u'xa1This, is a sentence with weirdxbb symbolsu2026 appearing everywherexbf'
    >>> expected = u'xa1 This , is a sentence with weird xbb symbols u2026 appearing everywhere xbf'
    >>> assert toktok.tokenize(text, return_str=True) == expected
    >>> toktok.tokenize(text) == [u'xa1', u'This', u',', u'is', u'a', u'sentence', u'with', u'weird', u'xbb', u'symbols', u'u2026', u'appearing', u'everywhere', u'xbf']
    True
    """
    # Replace non-breaking spaces with normal spaces.
    NON_BREAKING = re.compile(u"u00A0"), " "

    # Pad some funky punctuation.
    FUNKY_PUNCT_1 = re.compile(u'([،;؛¿!"])}»›”؟¡%٪°±©®।॥…])'), r" 1 "
    # Pad more funky punctuation.
    FUNKY_PUNCT_2 = re.compile(u'([({[“‘„‚«‹「『])'), r" 1 "
    # Pad En dash and em dash
    EN_EM_DASHES = re.compile(u'([–—])'), r" 1 "

    # Replace problematic character with numeric character reference.
    AMPERCENT = re.compile('& '), '&amp; '
    TAB = re.compile('t'), ' 	 '
    PIPE = re.compile('|'), ' | '

    # Pad numbers with commas to keep them from further tokenization. 
    COMMA_IN_NUM = re.compile(r'(?<!,)([,،])(?![,d])'), r' 1 '

    # Just pad problematic (often neurotic) hyphen/single quote, etc.
    PROB_SINGLE_QUOTES = re.compile(r"(['’`])"), r' 1 '
    # Group ` ` stupid quotes ' ' into a single token.
    STUPID_QUOTES_1 = re.compile(r" ` ` "), r" `` "
    STUPID_QUOTES_2 = re.compile(r" ' ' "), r" '' "

    # Don't tokenize period unless it ends the line and that it isn't 
    # preceded by another period, e.g.  
    # "something ..." -> "something ..." 
    # "something." -> "something ." 
    FINAL_PERIOD_1 = re.compile(r"(?<!.).$"), r" ."
    # Don't tokenize period unless it ends the line eg. 
    # " ... stuff." ->  "... stuff ."
    FINAL_PERIOD_2 = re.compile(r"""(?<!.).s*(["'’»›”]) *$"""), r" . 1"

    # Treat continuous commas as fake German,Czech, etc.: „
    MULTI_COMMAS = re.compile(r'(,{2,})'), r' 1 '
    # Treat continuous dashes as fake en-dash, etc.
    MULTI_DASHES = re.compile(r'(-{2,})'), r' 1 '
    # Treat multiple periods as a thing (eg. ellipsis)
    MULTI_DOTS = re.compile(r'(.{2,})'), r' 1 '

    # This is the p{Open_Punctuation} from Perl's perluniprops
    # see http://perldoc.perl.org/perluniprops.html
    OPEN_PUNCT = text_type(u'([{u0f3au0f3cu169bu201au201eu2045u207d'
                            u'u208du2329u2768u276au276cu276eu2770u2772'
                            u'u2774u27c5u27e6u27e8u27eau27ecu27eeu2983'
                            u'u2985u2987u2989u298bu298du298fu2991u2993'
                            u'u2995u2997u29d8u29dau29fcu2e22u2e24u2e26'
                            u'u2e28u3008u300au300cu300eu3010u3014u3016'
                            u'u3018u301au301dufd3eufe17ufe35ufe37ufe39'
                            u'ufe3bufe3dufe3fufe41ufe43ufe47ufe59ufe5b'
                            u'ufe5duff08uff3buff5buff5fuff62')
    # This is the p{Close_Punctuation} from Perl's perluniprops
    CLOSE_PUNCT = text_type(u')]}u0f3bu0f3du169cu2046u207eu208eu232a'
                            u'u2769u276bu276du276fu2771u2773u2775u27c6'
                            u'u27e7u27e9u27ebu27edu27efu2984u2986u2988'
                            u'u298au298cu298eu2990u2992u2994u2996u2998'
                            u'u29d9u29dbu29fdu2e23u2e25u2e27u2e29u3009'
                            u'u300bu300du300fu3011u3015u3017u3019u301b'
                            u'u301eu301fufd3fufe18ufe36ufe38ufe3aufe3c'
                            u'ufe3eufe40ufe42ufe44ufe48ufe5aufe5cufe5e'
                            u'uff09uff3duff5duff60uff63')
    # This is the p{Close_Punctuation} from Perl's perluniprops
    CURRENCY_SYM = text_type(u'$xa2xa3xa4xa5u058fu060bu09f2u09f3u09fb'
                             u'u0af1u0bf9u0e3fu17dbu20a0u20a1u20a2u20a3'
                             u'u20a4u20a5u20a6u20a7u20a8u20a9u20aau20ab'
                             u'u20acu20adu20aeu20afu20b0u20b1u20b2u20b3'
                             u'u20b4u20b5u20b6u20b7u20b8u20b9u20baua838'
                             u'ufdfcufe69uff04uffe0uffe1uffe5uffe6')

    # Pad spaces after opening punctuations.
    OPEN_PUNCT_RE = re.compile(u'([{}])'.format(OPEN_PUNCT)), r'1 '
    # Pad spaces before closing punctuations.
    CLOSE_PUNCT_RE = re.compile(u'([{}])'.format(CLOSE_PUNCT)), r'1 '
    # Pad spaces after currency symbols.
    CURRENCY_SYM_RE = re.compile(u'([{}])'.format(CURRENCY_SYM)), r'1 '

    # Use for tokenizing URL-unfriendly characters: [:/?#]
    URL_FOE_1 = re.compile(r':(?!//)'), r' : ' # in perl s{:(?!//)}{ : }g;
    URL_FOE_2 = re.compile(r'?(?!S)'), r' ? ' # in perl s{?(?!S)}{ ? }g;
    # in perl: m{://} or m{S+.S+/S+} or s{/}{ / }g;
    URL_FOE_3 = re.compile(r'(://)[S+.S+/S+][/]'), ' / '
    URL_FOE_4 = re.compile(r' /'), r' / ' # s{ /}{ / }g;

    # Left/Right strip, i.e. remove heading/trailing spaces.
    # These strip regexes should NOT be used,
    # instead use str.lstrip(), str.rstrip() or str.strip() 
    # (They are kept for reference purposes to the original toktok.pl code)  
    LSTRIP = re.compile(r'^ +'), ''
    RSTRIP = re.compile(r's+$'),'n' 
    # Merge multiple spaces.
    ONE_SPACE = re.compile(r' {2,}'), ' '

    TOKTOK_REGEXES = [NON_BREAKING, FUNKY_PUNCT_1, 
                      URL_FOE_1, URL_FOE_2, URL_FOE_3, URL_FOE_4,
                      AMPERCENT, TAB, PIPE,
                      OPEN_PUNCT_RE, CLOSE_PUNCT_RE, 
                      MULTI_COMMAS, COMMA_IN_NUM, FINAL_PERIOD_2,
                      PROB_SINGLE_QUOTES, STUPID_QUOTES_1, STUPID_QUOTES_2,
                      CURRENCY_SYM_RE, EN_EM_DASHES, MULTI_DASHES, MULTI_DOTS,
                      FINAL_PERIOD_1, FINAL_PERIOD_2, ONE_SPACE]

    def tokenize(self, text, return_str=False):
        text = text_type(text) # Converts input string into unicode.
        for regexp, subsitution in self.TOKTOK_REGEXES:
            text = regexp.sub(subsitution, text)
        # Finally, strips heading and trailing spaces
        # and converts output string into unicode.
        text = text_type(text.strip()) 
        return text if return_str else text.split()

Is there a way to make the subtituition faster? E.g.

  • Combine the chain of regexes into one super regex.
  • Combine some of the regexes
  • Coding it in Cython (but Cython regexes are slow, no?)
  • Running the regex substitution in Julia and wrapping Julia code in Python

The use case for the tokenize() function usually takes a single input but if the same function is called 1,000,000,000 times, it’s rather slow and the GIL is going to lock up the core and process each sentence at a time.

The aim of the question is to ask for ways to speed up a Python code that’s made up of regex substitution, esp. when running the tokenize() function for 1,000,000,000+ times.

If Cython/Julia or any faster language + wrapper is suggested, it would be good if you give an one regex example of how the regex is written in Cython/Julia/Others and the suggestion on how the wrapper would look like.


Get this bounty!!!

#StackBounty: #ssd #performance SanDisk SSD Plus: Half the performance on Linux than on Windows?

Bounty: 50

I have two SSDs in my laptop:

  • Crucial MX300 725GB –> /dev/sda
  • SanDisk SSD Plus 240GB –> /dev/sdb

They’re performance reads on Linux and Windows like this:

Crucial MX300 –> same on both OSs

sudo hdparm -tT /dev/sda # Crucial
Timing cached reads:   13700 MB in  2.00 seconds = 6854.30 MB/sec
Timing buffered disk reads: 1440 MB in  3.00 seconds = 479.58 MB/sec

Crucial MX300 725GB

SanDisk Plus –> way faster on Windows!

sudo hdparm -tT /dev/sdb # SanDisk
Timing cached reads:   7668 MB in  2.00 seconds = 3834.92 MB/sec
Timing buffered disk reads: 798 MB in  3.00 seconds = 265.78 MB/sec # TOO LOW !!

SanDisk

The sequential read performance of the SanDisk on Linux is about half of its performance on Windows!

My Question is of course: Why and can that be fixed? Is this due to the SanDisk SSD Plus being handled as a SCSI drive?

From syslog:

~$ grep SDSSD /var/log/syslog
systemd[1]: Found device SanDisk_SDSSDA240G
kernel: [    2.152138] ata2.00: ATA-9: SanDisk SDSSDA240G, Z32070RL, max UDMA/133
kernel: [    2.174689] scsi 1:0:0:0: Direct-Access     ATA      SanDisk SDSSDA24 70RL PQ: 0 ANSI: 5
smartd[1035]: Device: /dev/sdb [SAT], SanDisk SDSSDA240G, S/N:162783441004, WWN:5-001b44-4a404e4f0, FW:Z32070RL, 240 GB
smartd[1035]: Device: /dev/sdb [SAT], state read from /var/lib/smartmontools/smartd.SanDisk_SDSSDA240G-162783441004.ata.state
smartd[1035]: Device: /dev/sdb [SAT], state written to /var/lib/smartmontools/smartd.SanDisk_SDSSDA240G-162783441004.ata.state

Compared to the Crucial MX300 which has on linux almost the same performance as on Windows:

~$ grep MX300 /var/log/syslog
systemd[1]: Found device Crucial_CT750MX300SSD1
kernel: [    1.775520] ata1.00: ATA-10: Crucial_CT750MX300SSD1,  M0CR050, max UDMA/133
smartd[1035]: Device: /dev/sda [SAT], Crucial_CT750MX300SSD1, S/N:16251486AC40, WWN:5-00a075-11486ac40, FW:M0CR050, 750 GB
smartd[1035]: Device: /dev/sda [SAT], state read from /var/lib/smartmontools/smartd.Crucial_CT750MX300SSD1-16251486AC40.ata.state
smartd[1035]: Device: /dev/sda [SAT], state written to /var/lib/smartmontools/smartd.Crucial_CT750MX300SSD1-16251486AC40.ata.state

Any help is very welcome!

Edit:

The difference that hdparm is showing on Linux is very real. I created two identical directories, one in each of the two drives, each directory containing about 25Gb of files (36395 files), and ran the exact same hashdeep checksum creation script on both dirs (the script just creates a md5-checksum for every file in the test dirs and stores all the checksums in one single file). These are the results:

test-sandisk# time create-file-integrity-md5sums.sh .
real    1m49.000s
user    1m24.868s
sys 0m15.808s

test-mx300# time create-file-integrity-md5sums.sh .
real    0m54.180s
user    1m4.628s
sys 0m11.640s

Same test with a single 7Gb file:

test-sandisk# time create-file-integrity-md5sums.sh .
real    0m26.986s
user    0m19.168s
sys 0m3.232s


test-mx300# time create-file-integrity-md5sums.sh .
real    0m17.285s
user    0m16.248s
sys 0m1.368s


Get this bounty!!!