#StackBounty: #c++ #file-system Files manager api in c++

Bounty: 50

I’m new to this community, so I’m not sure if this should be a topic here, so just let me know if I’m not in the right place.

I wrote a file manager api in c++ to make the read-write operations easier in read/write cases, needing big structured data to/from multiple files. I want to get a review of the library if it’s readable and easy to understand, and if it serves its purpose.

File.hpp

#ifndef FILESAPI_FILE_H
#define FILESAPI_FILE_H

#include <iostream>
#include <mutex>
#include <fstream>
#include <vector>
#include <boost/assert.hpp>
#include "../utilities/DesignText.hpp"
#include "../utilities/Exceptions.hpp"

namespace FilesApi {

    /// Use in read/write for non-vectors overload operator. e.g: file >> rw_t<T>{val, 1};
    template<typename T> struct rw_s {
        T*          val;
        size_t      val_size;

        rw_s(T &value, size_t arr_size = 1) : val(&value), val_size(arr_size) {
            assert(arr_size > 0);

        }

        rw_s(T *value, size_t arr_size = 1) : val(value), val_size(arr_size) {
            assert(arr_size > 0);
            assert(value != nullptr);
        }
    };

    /// Wrapper function for creation of rw_t object, without need for specify the type after the function name
    /// Instead of call:    f << rw_t<int>(a, size);
    /// Call:               f << rw_soft(a, size);
    template<typename T> rw_s<T> rw_soft(T &value, size_t arr_size = 1) {
        return rw_s<T>(value, arr_size);
    }

    template<typename T> rw_s<T> rw_soft(T *value, size_t arr_size = 1) {
        return rw_soft(*value, arr_size);
    }

    /**
     * >> if file_mode is OPEN_IN_ACTION:
     * SINGLE_AND_DONE - read/write single time, and then close the file.
     * SINGLE_AND_MORE - read/write single time, but don't close the file yet. After single read/write the mode will automatic update to SINGLE_AND_DONE mode.
     * MULTIPLE - close the file only in programmer order, until then the file will be remain open.
     */
    enum class ReadWriteMode {
        SINGLE_AND_DONE,
        SINGLE_AND_MORE,
        MULTIPLE,
        DONE
    };

    enum class FileAction {
        READ,
        WRITE,
        NONE
    };

    enum class FileMode {
        ALWAYS_OPEN,
        OPEN_IN_ACTION
    };

    class File {
    private:
        bool is_ready;
        std::string name;
        std::string path;
        FileMode file_mode;
        ReadWriteMode read_write_mode;
        int multiple_times_left;
        FileAction file_action;
        std::mutex read_write_mutex;
        std::fstream file_ptr;
        bool is_open;
        std::ios_base::openmode read_flags;
        std::ios_base::openmode write_flags;
        bool use_exceptions;

        /**
         * Open file in specific format
         * param mode_flags - fstream.open() flags.
         * param new_file_action - Open purpose.
         */
        void open(std::ios_base::openmode mode_flags, const FileAction &new_file_action);

        /**
         * Auto update for the file's mode (READ / WRITE / NONE).
         */
        void update_rwm();

        /**
         * Close file
         * param automatic - Close request from API(true) or from User(false)
         */
        void close(bool automatic);

        /**
         * Is file ready for read/write actions. Exception if file not ready.
         * return Is file name not empty.
         */
        bool is_file_ready(int);

    public:
        /**
         * Ctor
         * param file_name     - if @param file_path == "" => path/to/file/filename.bin else filename.bin
         * param exceptions    - Throw exceptions on errors Or use bold cout messages.
         * param file_path     - file's path.
         */
        explicit File(const std::string &file_name, bool exceptions = false, const std::string &file_path = "");

        /**
         * Close the file.
         */
        ~File();

        /**
         * Force close the file.
         */
        void close();

        /**
         * Set file's name.
         * param new_name - New file's name.
         */
        void set_name(const std::string &new_name);

        /**
         * Set file's name.
         * param new_name - New file's name.
         */
        File &operator=(const std::string &new_name);

        /**
         * Get file's name
         * return File's name.
         */
        std::string get_name();

        /**
         * Is file ready for read/write actions. Without exception if file not ready.
         * return Is file name not empty.
         */
        bool is_file_ready();

        /**
         * Init current file's mode
         * param mode - How much reads/writes until the file will close.
         * param multiple_times - if mode is multiple note how much times (-1 for unknown - won't close the file without programmer order/interrupt).
         */
        void init_read_write_mode(const ReadWriteMode &mode, int multiple_times = -1);

        /**
         * Init read fstream flags.
         * param read_flags - When open function in read mode will occur, those flags will be in use.
         */
        void init_read_flags(std::ios_base::openmode read_flags = std::ios_base::in);

        /**
         * Init write fstream flags.
         * param write_flags - When open function in write mode will occur, those flags will be in use.
         */
        void init_write_flags(std::ios_base::openmode write_flags = std::ios::out | std::ios::binary | std::ios::in);

        /**
         * Read to non-vector variable
         * param T - variable type
         * param val - variable address
         * param data_size - in case of array- array's size.
         * return this File object.
         */
        template<class T>
        File &read(T *val, size_t data_size = 1);

        /**
         * Read to vector variable
         * tparam T - vector type
         * param val - vector to read into (Have to be initialize with the size of inputs' count).
         * param data_size - vector to read into (Have to be initialize with the size of inputs' count).
         * return this File object.
         */
        template<class T>
        File &read(std::vector<T> &val);

        /**
         * Write non-vector variable
         * tparam T - variable type
         * param val - variable address
         * param data_size - in case of array- array's size.
         * return this File object.
         */
        template<class T>
        File &write(const T *val, size_t data_size = 1);

        /**
         * Write vector variable
         * tparam T - vector type
         * param val - vector to write.
         * return this File object.
         */
        template<class T>
        File &write(const std::vector<T> &val);

        /**
         * Read to vector
         * tparam T - vector type
         * param data - vector to read into
         * return this File object.
         */
        template<class T>
        File &operator>>(std::vector<T> &data);

        /**
         * Read to non-vector
         * tparam T - variable type
         * param info - {
         *                  val - variable non-vector to read into
         *                  val_size - in case of array- array's size (else leave as default 1)
         *               }
         * return this File object
         */
        template<class T>
        File &operator>>(const rw_s<T> &info);

        /**
         * Write vector to file
         * tparam T - vector type
         * param data - vector to write
         * return this File object
         */
        template<class T>
        File &operator<<(const std::vector<T> &data);

        /**
         * Write non-vector to file
         * tparam T - variable type
         * param info - {
         *                  val - variable non-vector to write
         *                  val_size - in case of array- array's size (else leave as default 1)
         *               }
         * return this File object
         */
        template<class T>
        File &operator<<(const rw_s<T> &info);
    };

    template<class T>
    File &File::read(T *val, const size_t data_size) {
        if (!is_file_ready(0)) {
            return *this;
        }
        open(read_flags, FileAction::READ);
        std::lock_guard<std::mutex> guard(read_write_mutex);

        file_ptr.read((char *) (val), sizeof(T) * data_size);

        update_rwm();
        return *this;
    }

    template<class T>
    File &File::write(const T *val, const size_t data_size) {
        if (!is_file_ready(0)) {
            return *this;
        }
        open(write_flags, FileAction::WRITE);
        std::lock_guard<std::mutex> guard(read_write_mutex);

        file_ptr.write(reinterpret_cast<const char *>(val), sizeof(T) * data_size);

        update_rwm();
        return *this;
    }

    template<class T>
    File &File::read(std::vector<T> &val) {
        if (!is_file_ready(0)) {
            return *this;
        }
        open(read_flags, FileAction::READ);
        std::lock_guard<std::mutex> guard(read_write_mutex);

        file_ptr.read(reinterpret_cast<char *>(val.data()), sizeof(T) * val.size());

        update_rwm();
        return *this;
    }

    template<typename T>
    File &File::write(const std::vector<T> &val) {
        if (!is_file_ready(0)) {
            return *this;
        }
        open(write_flags, FileAction::WRITE);
        std::lock_guard<std::mutex> guard(read_write_mutex);

        file_ptr.write(reinterpret_cast<const char *>(val.data()), sizeof(T) * val.size());

        update_rwm();
        return *this;
    }

    template<class T>
    File &File::operator>>(std::vector<T> &data) {
        return read(data);
    }

    template<class T>
    File &File::operator>>(const rw_s<T> &info) {
        return read(info.val, info.val_size);
    }

    template<class T>
    File &File::operator<<(const std::vector<T> &data) {
        return write(data);
    }

    template<class T>
    File &File::operator<<(const rw_s<T> &info) {
        return write(info.val, info.val_size);
    }
}

#endif //FILESAPI_FILE_H

FilesManager.hpp

#ifndef FILESAPI_FILESMANAGER_H
#define FILESAPI_FILESMANAGER_H

#include <iostream>
#include <vector>
#include <map>
#include <memory>
#include "File.hpp"

namespace FilesApi {
    using add_data = std::tuple<const std::string, const std::string>;

    class FilesManager {
    private:
        std::map<std::string, std::shared_ptr<File>> files;
        size_t max_files; // zero for unlimited
        std::string files_path; // Leave "" if there is no single path for all of the files
        bool use_exceptions;

        void remove_unusable_files();

    public:
        /**
         * Ctor
         * param exceptions - Throw exceptions on errors Or use bold cout messages.
         * param max_files - Maximum files number to manage in this FilesManager object (0 for unlimited).
         * param files_path - if @param files_path == "" => in new file associate you will have to supply full
         *                  file path, e.g: "path/to/file/filename.bin"
         *                  else supply only file name, e.g: "filename.bin", if @param files_path == "path/to/file/"
         */
        explicit FilesManager(bool exceptions = false, size_t max_files = 0, const std::string &files_path = "");

        /**
         * Add new file
         * param id - file id (will be use to get this File object).
         * param file - file's name or path (if @files_path == "").
         */
        void add(const std::string &id, const std::string &file);

        /**
         * Remove file
         * param id - file's id
         */
        void remove(const std::string &id);

        /**
         * Get file
         * param id - file's id
         */
        File &get(const std::string &id);

        /**
         * Add new file
         * param data - tuple(0) => file id. tuple(1_ => file name or path (if files path is "").
         */
        void operator+=(const add_data &data);

        /**
         * Get file
         * param id - file's id
         */
        File &operator[](const std::string &id);

        /**
         * Remove file
         * param id - file's id
         */
        void operator-=(const std::string &id);
    };
}
#endif //FILESAPI_FILESMANAGER_H

File.cpp

#include "../headers/File.hpp"

namespace FilesApi {
    File::File(const std::string &file_name, bool exceptions, const std::string &file_path) {
        name = file_name;
        path = file_path;
        is_ready = !name.empty();
        use_exceptions = exceptions;
        is_open = false;
        file_mode = FileMode::OPEN_IN_ACTION;
        read_write_mode = ReadWriteMode::DONE;
        file_action = FileAction::NONE;
        write_flags = std::ios::out | std::ios::binary | std::ios::in;
        read_flags = std::ios_base::in;
    }

    File::~File() {
        std::lock_guard<std::mutex> guard(read_write_mutex);
        close();
    }

    void File::open(std::ios_base::openmode mode_flags, const FileAction &new_file_action) {
        std::lock_guard<std::mutex> guard(read_write_mutex);
        if (!is_file_ready(0)) {
            if (is_open) {
                file_ptr.close();
                is_open = false;
            }
            return;
        }
        if (file_action != new_file_action) {
            file_ptr.close();
            is_open = false;
            if (file_action != FileAction::NONE) {
                std::cout
                        << DesignText::make_colored("Pay attention: file mission replaced by another one. (file closed)",
                                                    DesignText::Color::RED, false) << std::endl;
            }
        }
        file_action = new_file_action;

        if (!is_open) {
            file_ptr.open(path + name, mode_flags);
            if (file_ptr.fail()) {
                is_open = false;
                if (!use_exceptions) {
                    std::cout << DesignText::make_colored("Error Opening file: " + path + name,
                                                        DesignText::Color::RED, true) << std::endl;
                } else {
                    throw FileOpenException(path + name);
                }
            }
            is_open = true;
            std::cout << DesignText::make_colored("File has safely opened.", DesignText::Color::GREEN, false)
                      << std::endl;
        }
    }

    void File::close(bool automatic) {
        if ((!automatic) || (file_mode == FileMode::OPEN_IN_ACTION)) {
            if (is_open) {
                file_ptr.close();
                is_open = false;
                read_write_mode = ReadWriteMode::DONE;
                file_action = FileAction::NONE;
                std::cout << DesignText::make_colored("File has safely closed.", DesignText::Color::GREEN, false)
                          << std::endl;
            }
        }
    }

    void File::close() {
        close(false);
    }

    void File::update_rwm() {
        if (!is_file_ready(0)) {
            return;
        }
        switch (read_write_mode) {
            case ReadWriteMode::SINGLE_AND_DONE:
            case ReadWriteMode::DONE:
                close(true);
                break;
            case ReadWriteMode::SINGLE_AND_MORE:
                read_write_mode = ReadWriteMode::SINGLE_AND_DONE;
                break;
            case ReadWriteMode::MULTIPLE:
                if (multiple_times_left > -1 && !--multiple_times_left) {
                    multiple_times_left = -1;
                    close(true);
                }
                break;
        }
    }

    void File::init_read_write_mode(const ReadWriteMode &new_mode, const int multiple_times) {
        read_write_mode = new_mode;
        multiple_times_left = multiple_times;
    }

    void File::init_read_flags(const std::ios_base::openmode new_read_flags) {
        read_flags = new_read_flags;
    }

    void File::init_write_flags(const std::ios_base::openmode new_write_flags) {
        write_flags = new_write_flags;
    }

    void File::set_name(const std::string &new_name) {
        if (!new_name.empty()) {
            name = new_name;
            is_ready = true;
            return;
        }
        if (name.empty()) {
            is_ready = false;
        }
    }

    std::string File::get_name() {
        return name;
    }

    bool File::is_file_ready(int) {
        if (!is_ready) {

            if (!use_exceptions) {
                std::cout << DesignText::make_colored("Pay attention: file name is empty. can't open this file.",
                                                    DesignText::Color::RED, true) << std::endl;
            } else {
                throw FileNotReadyException();
            }
            return false;
        }
        return true;
    }

    bool File::is_file_ready() {
        return is_ready;
    }

    File &File::operator=(const std::string &new_name) {
        set_name(new_name);
        return *this;
    }
}

FilesManager.cpp

#include "../headers/FilesManager.hpp"

namespace FilesApi {
    FilesManager::FilesManager(bool exceptions, size_t max_files, const std::string &files_path)
            : max_files(max_files), files_path(files_path), use_exceptions(exceptions) {
    }

    void FilesManager::add(const std::string &id, const std::string &file) {
        remove_unusable_files();
        if (max_files == 0 || files.size() + 1 < max_files) {
            files.insert(std::pair<std::string,
                    std::shared_ptr<File>>(id, std::make_shared<File>(file, use_exceptions, files_path)));
        }
    }

    void FilesManager::remove(const std::string &id) {
        remove_unusable_files();
        files.erase(id);
    }

    File &FilesManager::get(const std::string &id) {
        remove_unusable_files();
        File *ret_file = files[id].get();
        if (ret_file == nullptr) {
            files[id] = std::make_shared<File>("", use_exceptions, files_path);
            ret_file = files[id].get();
        }
        return *ret_file;
    }

    void FilesManager::operator+=(const add_data &data) {
        add(std::get<0>(data), std::get<1>(data));
    }

    File &FilesManager::operator[](const std::string &id) {
        return get(id);
    }

    void FilesManager::operator-=(const std::string &id) {
        remove(id);
    }

    void FilesManager::remove_unusable_files() {
        for (auto &file : files) {
            if (file.second && !file.second->is_file_ready()) {
                files.erase(file.first);
            }
        }
    }
}

Use example:

#include <iostream>
#include <vector>
#include <complex>
#include "../src/headers/FilesManager.hpp"

using namespace std;
using namespace FilesApi;

int mainFilesManagerOperatorsTest() {
    FilesManager fm(false, 0, "../TestFiles/");
    string files[] = {"test_file.bin", "test_file2.bin"};

    fm["1"] = files[0];

    vector<complex<float>> wdata = {{1, 9}, {3, 75}, {213.34, 21.4}, {153.1, 15.85}};
    vector<complex<float>> rdata(wdata.size());

    fm["1"].init_read_write_mode(ReadWriteMode::SINGLE_AND_DONE);
    //fm.get("1").write(wdata.data(), wdata.size()); // Use it as complex<float> array. Possible.
    fm["1"].write(wdata);
    fm["1"].init_read_write_mode(ReadWriteMode::SINGLE_AND_DONE);
    fm["1"].read(rdata);

    fm += add_data("5", files[1]); // Add file to collection
    int a = 12;
    int b;
    fm["5"] << rw_soft(a); // Work
    fm["5"].write(&a); // Work
    fm["5"] >> rw_soft(b); // Work
    cout << b << endl; // Prints 12

    fm -= "5"; // Remove the file from collection
    fm["5"] << rw_soft(a); // Error
    fm["5"].write(&a); // Error
    fm["5"] >> rw_soft(b); // Error

    //fm["2"] = files[1];
    fm += add_data("2", files[1]);

    for (size_t i = 0; i < rdata.size(); i++) {
        cout << rdata[i] << endl;
    }


    fm["2"].init_read_write_mode(ReadWriteMode::MULTIPLE);
    for (size_t i = 0; i < 100; i++) {
        fm["2"].write(&i);
    }

    //f.init_read_write_mode(ReadWriteMode::MULTIPLE);

    size_t j;
    for (size_t i = 0; i < 100; i++) {
        fm["2"].read(&j);
        cout << j << " ";
    }
    cout << endl;

    return 0;
}

You can find more examples/code utilities in Github: https://github.com/korelkashri/filesApi


Get this bounty!!!

#StackBounty: #performance #beginner #php #file-system #api Equity data processing: Fast and/or efficient file writing using PHP

Bounty: 50

Problem

This is my first scripting project and I’m sure it has so many issues.

The main class, EQ, scrapes equities data using EquityRecords, calculates sector coefficients using
SectorMovers, estimates equity prices, and finally writes ~8,000 HTML strings (~100Kb-120Kb) on .md files for viewing.

I couldn’t add EQ code in the post due to character limitation of the posts. Please see the entire code on this GitHub link.


Performance

The only goal is making EQ as fast/efficient as possible for a single server. Would you be so kind and review it and kindly help me to reach this goal?


EquityRecords

date_default_timezone_set("UTC");
ini_set('max_execution_time', 0);
ini_set('memory_limit', '-1');
set_time_limit(0);

// EquityRecords::allEquitiesSignleJSON(new EquityRecords());

class EquityRecords
{

    const NUMBER_OF_STOCKS_PER_REQUEST = 100;
    const NEW_LINE = "n";

    /**
     *
     * @var a string of iextrading symbols
     */

    const SYMBOLS_PATH = '/../../config/z-iextrading-symbs.md';

    /**
     *
     * @var a string of our symbols json directory
     */

    const SYMBOLS_DIR = "/../../blog-back/equities/real-time-60sec/z-raw-equilibrium-estimation";

    /**
     *
     * @var a string of target path and query
     */

    const TARGET_QUERY = "stock/market/batch?symbols=";

    /**
     *
     * @var a string of iextrading base URL
     */

    const BASE_URL = "https://api.iextrading.com/1.0/";

    /**
     *
     * @var a string of iextrading end point
     */

    const END_POINT = "&types=quote,chart&range=1m&last=10";

    /**
     *
     * @var an integer for maximum number of stocks per URL on each call
     */

    //***************** A ********************** //
    // public static function getSymbols() {
    //     return array_map(function($line){ return str_getcsv($line, "t"); }, file(__DIR__ . self::SYMBOLS_PATH));
    // }

    public static function getSymbols()
    {

        //***************** START: ALL SYMBOLS ARRAY ********************** //
        // var: is a filename path directory, where there is an md file with list of equities
        $list_of_equities_file = __DIR__ . self::SYMBOLS_PATH;

        // var: is content of md file with list of equities
        $content_of_equities = file_get_contents($list_of_equities_file);

        // var is an array(3) of equities such as: string(4) "ZYNE", string(10) "2019-01-04", string(27) "ZYNERBA PHARMACEUTICALS INC"

        // $symbols_array=preg_split('/rn|r|n/', $content_of_equities);
        $symbols_array = preg_split('/R/', $content_of_equities);
        //***************** END: ALL SYMBOLS ARRAY ********************** //

        // child and mother arrays are created to help calling equities in batches of 100, which seems to be the API limit.
        $child = array();
        $mother = array();
        // var: is 100 counter
        $limit_counter = self::NUMBER_OF_STOCKS_PER_REQUEST;
        foreach ($symbols_array as $ticker_arr) {
            $limit_counter = $limit_counter - 1;
            $symbols_array = preg_split('/t/', $ticker_arr);
            array_push($child, $symbols_array);

            if ($limit_counter <= 0) {
                $limit_counter = self::NUMBER_OF_STOCKS_PER_REQUEST;
                array_push($mother, $child);
                $child = array();
            }

        }
        return $mother;
    }

    public static function allEquitiesSignleJSON()
    {
        $equity_arrays = EquityRecords::getSymbols();
        $base_url = self::BASE_URL . self::TARGET_QUERY;

        $current_time = date("Y-m-d-H-i-s");
        $all_equities = array();
        // ticker: AAPL, GE, AMD
        foreach ($equity_arrays as $ticker_arr) {
            $ticker = array_column($ticker_arr, 0);
            $equity_url = $base_url . implode("%2C", $ticker) . self::END_POINT;
            $raw_eauity_json = file_get_contents($equity_url);
            $raw_equity_array = json_decode($raw_eauity_json, true);
            $all_equities = array_merge($all_equities, $raw_equity_array);
        }

        $all_equities_json = json_encode($all_equities);

        $symbols_dir = __DIR__ . self::SYMBOLS_DIR;

        if (!is_dir($symbols_dir)) {mkdir($symbols_dir, 0755, true);}

        $raw_equity_file = $symbols_dir . "/" . $current_time . ".json";
        $fp = fopen($raw_equity_file, "x+");
        fwrite($fp, $all_equities_json);
        fclose($fp);
        echo "YAAAY! Equity JSON file success at " . __METHOD__ . " ! 💚 " . self::NEW_LINE;
    }

    /**
     * @return a string for var_dump
     */
    public static function p()
    {
        $args = func_get_args();
        $die = (end($args) === 1) && array_pop($args);
        echo self::NEW_LINE;
        foreach ($args as $v) {
            $output = print_r($v, true);
            var_dump($output) . self::NEW_LINE;
        }
        echo self::NEW_LINE;
        if ($die) {
            die();
        }

    }

}

SectorMovers

date_default_timezone_set("UTC");
ini_set('max_execution_time', 0);
ini_set('memory_limit', '-1');
set_time_limit(0);

require_once __DIR__ . "/EquityRecords.php";

SectorMovers::getSectors();

class SectorMovers
{

    /**
     *
     * @var a string of iextrading base URL
     */

    const BASE_URL = "https://api.iextrading.com/1.0/";

    /**
     *
     * @var a string of target path and query
     */

    const TARGET_QUERY = "stock/market/batch?symbols=";

    /**
     *
     * @var a string for backend path for every sector
     */

    const EACH_SECTOR_DIR_PREFIX = "/../../blog-back/sectors/real-time-60sec/z-raw-sector-";

    /**
     *
     * @var a string for backend path for index sector
     */

    const INDEX_SECTOR_DIR_PREFIX = "/../../blog-back/sectors/real-time-60sec/y-index/";

    /**
     *
     * @var a string for live data path
     */

    const LIVE_DATA_DIR = "/../../../public_html/blog/files/";
    const DIR_FRONT_SECTOR_COEF_FILENAME = "s-1.txt"; // Filename that records sector coefficient JSON

    public static function getSectors()
    {
        $base_url = self::BASE_URL . self::TARGET_QUERY;
        $current_time = date("Y-m-d-H-i-s");

        $permission = 0755;

        $index_data = array("Overall" => array("sector_weight" => 1, "sector_coefficient" => 1, "sector_value" => 0));
        $sector_movers = SectorMovers::iexSectorParams();
        foreach ($sector_movers as $sector_mover) {
            // $sector_url = $base_url . implode(",", array_keys($sector_mover["selected_tickers"])) . "&types=quote&range=1m";
            $sector_url = $base_url . implode("%2C", array_keys($sector_mover["selected_tickers"])) . "&types=quote&range=1m";
            $rawSectorJson = file_get_contents($sector_url);
            $raw_sector_array = json_decode($rawSectorJson, true);

            // ******************* Back Data ***************** //
            // Write the raw file in the back directories

            // $rawSectorDir =  __DIR__ . self::EACH_SECTOR_DIR_PREFIX . $sector_mover["directory"];

            // // if back directory not exist
            // if (!is_dir($rawSectorDir)) {mkdir($rawSectorDir, $permission, true);}

            // // create and open/write/close sector data to back directories
            // $rawSectorFile = $rawSectorDir . "/" . $current_time . ".json";
            // $fp = fopen($rawSectorFile, "a+");
            // fwrite($fp, $rawSectorJson);
            // fclose($fp);

            // ******************* End Back Data ***************** //

            // Calculate the real-time index
            $index_value = 0;
            foreach ($raw_sector_array as $ticker => $ticker_stats) {
                if (isset($sector_mover["selected_tickers"][$ticker], $ticker_stats["quote"], $ticker_stats["quote"]["extendedChangePercent"], $ticker_stats["quote"]["changePercent"], $ticker_stats["quote"]["ytdChange"])) {

                    $change_amount = ($ticker_stats["quote"]["extendedChangePercent"] + $ticker_stats["quote"]["changePercent"] + $ticker_stats["quote"]["ytdChange"]) / 200;
                    $index_value += $sector_mover["sector_weight"] * $sector_mover["selected_tickers"][$ticker] * $change_amount;
                }
            }

            $index_data[$sector_mover["sector"]] = array("sector_weight" => $sector_mover["sector_weight"], "sector_coefficient" => $sector_mover["sector_coefficient"], "sector_value" => $index_value);
            $index_data["Overall"]["sector_value"] += $index_data[$sector_mover["sector"]]["sector_value"];
        }

        // Calculate the index factor for better visibility between -1 and +1
        $front_index_data = array();
        foreach ($index_data as $sector_name => $sector_index_data) {

            // $index_sign = $sector_index_data["sector_value"];
            // if ($index_sign < 0) {
            //     $index_sign = - $index_sign;
            // }

            $index_sign = abs($sector_index_data["sector_value"]);

            $index_factor = 1;
            for ($i = 0; $i <= 10; $i++) {
                $index_factor = pow(10, $i);
                if (($index_factor * $index_sign) > 1) {
                    $index_factor = pow(10, $i - 1);
                    break;
                }
            }

            // $index_factor = 10 ** strlen(preg_match('~.K0+~', $float, $zeros) ? $zeros[0] : 0);

            $front_index_data[$sector_name] = $sector_index_data["sector_weight"] * $sector_index_data["sector_coefficient"] * $sector_index_data["sector_value"] * $index_factor;
        }

        // Write the index file
        $index_sector_dir = __DIR__ . self::INDEX_SECTOR_DIR_PREFIX;

        if (!is_dir($index_sector_dir)) {mkdir($index_sector_dir, $permission, true);}

        $index_sector_file = $index_sector_dir . $current_time . ".json";

        $index_sector_json = json_encode($front_index_data, JSON_FORCE_OBJECT);
        $fp = fopen($index_sector_file, "a+");
        fwrite($fp, $index_sector_json);
        fclose($fp);

        $sector_dir = __DIR__ . self::LIVE_DATA_DIR;

        if (!is_dir($sector_dir)) {mkdir($sector_dir, $permission, true);} // if data directory did not exist

        // if s-1 file did not exist
        if (!file_exists($sector_dir . self::DIR_FRONT_SECTOR_COEF_FILENAME)) {
            $handle = fopen($sector_dir . self::DIR_FRONT_SECTOR_COEF_FILENAME, "wb");
            fwrite($handle, "d");
            fclose($handle);
        }

        $sector_coef_file = $sector_dir . self::DIR_FRONT_SECTOR_COEF_FILENAME;
        copy($index_sector_file, $sector_coef_file);
        echo "YAAAY! " . __METHOD__ . " updated sector coefficients successfully 💚!n";

        return $front_index_data;
    }

    public static function iexSectorParams()
    {
        $sector_movers = array(
            array(
                "sector" => "IT",
                "directory" => "information-technology",
                "sector_weight" => 0.18,
                "sector_coefficient" => 4,
                "selected_tickers" => array(
                    "AAPL" => 0.18,
                    "AMZN" => 0.16,
                    "GOOGL" => 0.14,
                    "IBM" => 0.2,
                    "MSFT" => 0.1,
                    "FB" => 0.1,
                    "NFLX" => 0.08,
                    "ADBE" => 0.06,
                    "CRM" => 0.04,
                    "NVDA" => 0.02,
                ),
            ),
            array(
                "sector" => "Telecommunication",
                "directory" => "telecommunication-services",
                "sector_weight" => 0.12,
                "sector_coefficient" => 4,
                "selected_tickers" => array(
                    "VZ" => 0.18,
                    "CSCO" => 0.16,
                    "CMCSA" => 0.14,
                    "T" => 0.12,
                    "CTL" => 0.1,
                    "CHTR" => 0.1,
                    "S" => 0.08,
                    "DISH" => 0.06,
                    "USM" => 0.04,
                    "VOD" => 0.02,
                ),
            ),
            array(
                "sector" => "Finance",
                "directory" => "financial-services",
                "sector_weight" => 0.1,
                "sector_coefficient" => 6,
                "selected_tickers" => array(
                    "JPM" => 0.18,
                    "GS" => 0.16,
                    "V" => 0.14,
                    "BAC" => 0.12,
                    "AXP" => 0.1,
                    "WFC" => 0.1,
                    "USB" => 0.08,
                    "PNC" => 0.06,
                    "AMG" => 0.04,
                    "AIG" => 0.02,
                ),
            ),
            array(
                "sector" => "Energy",
                "directory" => "energy",
                "sector_weight" => 0.1,
                "sector_coefficient" => 6,
                "selected_tickers" => array(
                    "CVX" => 0.18,
                    "XOM" => 0.16,
                    "APA" => 0.14,
                    "COP" => 0.12,
                    "BHGE" => 0.1,
                    "VLO" => 0.1,
                    "APC" => 0.08,
                    "ANDV" => 0.06,
                    "OXY" => 0.04,
                    "HAL" => 0.02,
                ),
            ),
            array(
                "sector" => "Industrials",
                "directory" => "industrials",
                "sector_weight" => 0.08,
                "sector_coefficient" => 8,
                "selected_tickers" => array(
                    "CAT" => 0.18,
                    "FLR" => 0.16,
                    "GE" => 0.14,
                    "JEC" => 0.12,
                    "JCI" => 0.1,
                    "MAS" => 0.1,
                    "FLS" => 0.08,
                    "AAL" => 0.06,
                    "AME" => 0.04,
                    "CHRW" => 0.02,
                ),
            ),
            array(
                "sector" => "Materials and Chemicals",
                "directory" => "materials-and-chemicals",
                "sector_weight" => 0.08,
                "sector_coefficient" => 8,
                "selected_tickers" => array(
                    "DWDP" => 0.18,
                    "APD" => 0.16,
                    "EMN" => 0.14,
                    "ECL" => 0.12,
                    "FMC" => 0.1,
                    "LYB" => 0.1,
                    "MOS" => 0.08,
                    "NEM" => 0.06,
                    "PPG" => 0.04,
                    "MLM" => 0.02,
                ),
            ),
            array(
                "sector" => "Utilities",
                "directory" => "utilities",
                "sector_weight" => 0.08,
                "sector_coefficient" => 8,
                "selected_tickers" => array(
                    "PPL" => 0.18,
                    "PCG" => 0.16,
                    "SO" => 0.14,
                    "WEC" => 0.12,
                    "PEG" => 0.1,
                    "XEL" => 0.1,
                    "D" => 0.08,
                    "NGG" => 0.06,
                    "NEE" => 0.04,
                    "PNW" => 0.02,
                ),
            ),
            array(
                "sector" => "Consumer Discretionary",
                "directory" => "consumer-discretionary",
                "sector_weight" => 0.08,
                "sector_coefficient" => 8,
                "selected_tickers" => array(
                    "DIS" => 0.18,
                    "HD" => 0.16,
                    "BBY" => 0.14,
                    "CBS" => 0.12,
                    "CMG" => 0.1,
                    "MCD" => 0.1,
                    "GPS" => 0.08,
                    "HOG" => 0.06,
                    "AZO" => 0.04,
                    "EXPE" => 0.02,
                ),
            ),
            array(
                "sector" => "Consumer Staples",
                "directory" => "consumer-staples",
                "sector_weight" => 0.06,
                "sector_coefficient" => 8,
                "selected_tickers" => array(
                    "PEP" => 0.18,
                    "PM" => 0.16,
                    "PG" => 0.14,
                    "MNST" => 0.12,
                    "TSN" => 0.1,
                    "CPB" => 0.1,
                    "HRL" => 0.08,
                    "SJM" => 0.06,
                    "CAG" => 0.04,
                    "KHC" => 0.02,
                ),
            ),
            array(
                "sector" => "Defense",
                "directory" => "defense-and-aerospace",
                "sector_weight" => 0.04,
                "sector_coefficient" => 10,
                "selected_tickers" => array(
                    "BA" => 0.18,
                    "LMT" => 0.16,
                    "UTX" => 0.14,
                    "NOC" => 0.12,
                    "HON" => 0.1,
                    "RTN" => 0.1,
                    "TXT" => 0.08,
                    "LLL" => 0.06,
                    "COL" => 0.04,
                    "GD" => 0.02,
                ),
            ),
            array(
                "sector" => "Health",
                "directory" => "health-care-and-pharmaceuticals",
                "sector_weight" => 0.04,
                "sector_coefficient" => 10,
                "selected_tickers" => array(
                    "UNH" => 0.18,
                    "JNJ" => 0.16,
                    "PFE" => 0.14,
                    "UHS" => 0.12,
                    "AET" => 0.1,
                    "RMD" => 0.1,
                    "TMO" => 0.08,
                    "MRK" => 0.06,
                    "ABT" => 0.04,
                    "LLY" => 0.02,
                ),
            ),

            array(
                "sector" => "Real Estate",
                "directory" => "real-estate",
                "sector_weight" => 0.04,
                "sector_coefficient" => 10,
                "selected_tickers" => array(
                    "CCI" => 0.18,
                    "AMT" => 0.16,
                    "AVB" => 0.14,
                    "HCP" => 0.12,
                    "RCL" => 0.1,
                    "HST" => 0.1,
                    "NCLH" => 0.08,
                    "HLT" => 0.06,
                    "ARE" => 0.04,
                    "AIV" => 0.02,
                ),
            ),
        );
        return $sector_movers;
    }

}

Acknowledgement

I’d like to thank these users for being so helpful, which I could implement some of their advices in the code.


Get this bounty!!!

#StackBounty: #performance #beginner #php #file-system #api API equity scraping: Fast and/or efficient file writing using PHP

Bounty: 50

This is my first scripting project, and the only goal is making the scripts as fast/efficient as possible (performance) for a single machine. Would you be so kind and review my working codes and kindly help me with the goal?

There is only one main class (EQ) that does this job, by scraping equities data using EquityRecordsclass and calculating sector coefficients using
SectorMovers class.

I tried to add EQ class in the post. However, I couldn’t because of character limitations of the post. You can view it in this GitHub link.

Acknowledgment

I’d also like to thank the following users for their kind helps:

  • Andreas

  • ArtisticPhoenix

  • Dharman

  • kemicofa

  • Mick Harner

  • Nina Scholz

  • Oh My Goodness

  • Sᴀᴍ Onᴇᴌᴀ


Get this bounty!!!