SimpleDataset is a sample implementation of a Dataset, where the data is stored as JPEG images in a single folder called images/ and features are stored in a folder feats/<feat_name>. More...

#include <dataset.hpp>

Inheritance diagram for SimpleDataset:

[legend]

Collaboration diagram for SimpleDataset:

[legend]

Data Structures
class	SimpleImage
	SimpleImage class used with the SimpleDataset class. More...

Public Member Functions
	SimpleDataset (const std::string &base_location)
	Creates a simple dataset from the images in base_location/images. More...

	SimpleDataset (const std::string &base_location, const std::string &db_data_location)
	If a dataset file is location at db_data_location, will load that file from. More...

	~SimpleDataset ()

bool	write (const std::string &db_data_location)
	Writes the SimpleDataset out to the specified file. More...

bool	read (const std::string &db_data_location)
	Reads the specified SimpleDataset. More...

std::shared_ptr< Image >	image (uint64_t id) const
	Given a unique integer ID, returns an Image associated with that ID. More...

bool	add_image (const std::shared_ptr< const Image > &image)
	Adds the given image to the database, if there is an id collision, will not add the image and return false, otherwise returns true. More...

uint64_t	num_images () const
	Returns the number of images in the dataset. More...

Public Member Functions inherited from Dataset
	Dataset (const std::string &base_location)
	Constructs a dataset given a base location. More...

	Dataset (const std::string &base_location, const std::string &db_data_location)
	Loads a dataset from the db_data_location. More...

virtual	~Dataset ()

std::string	location () const
	Returns the absolute path of the data directory. More...

std::string	location (const std::string &relative_path) const
	Returns the absolute path of the file (appends the file path to the database path). More...

std::vector< std::shared_ptr < const Image > >	all_images () const
	Returns a vector of all images in the dataset. More...

std::vector< std::shared_ptr < const Image > >	random_images (size_t count) const
	Returns a vector of random images in the dataset of size count. More...

std::vector< Dataset >	shard (const std::vector< std::string > &new_locations)
	: Shards the dataset to the new input locations, and returns the sharded datasets More...

Private Member Functions
void	construct_dataset ()
	Constructs the dataset an fills in the image id map. More...

Private Attributes
boost::bimap< std::string, uint64_t >	id_image_map

Additional Inherited Members
Protected Attributes inherited from Dataset
std::string	data_directory

Detailed Description

SimpleDataset is a sample implementation of a Dataset, where the data is stored as JPEG images in a single folder called images/ and features are stored in a folder feats/<feat_name>.

For example, given a base absolute path of /c/data/. Image data is found in /c/data/images and sift features are found in /c/data/feats/sift/.

Definition at line 76 of file dataset.hpp.

Constructor & Destructor Documentation

SimpleDataset::SimpleDataset ( const std::string & base_location )

Creates a simple dataset from the images in base_location/images.

It is recommended to then call write(...) to save the dataset so that it does not have to traverse the HDD everytime we load the dataset.

Definition at line 46 of file dataset.cxx.

References construct_dataset().

                                                            : Dataset(base_location) { 
         this->construct_dataset();
 }

SimpleDataset::SimpleDataset	(	const std::string &	base_location,
		const std::string &	db_data_location
	)

If a dataset file is location at db_data_location, will load that file from.

Otherwise, this will create the dataset from base_location/images and call write(db_data_location).

Definition at line 50 of file dataset.cxx.

References construct_dataset(), filesystem::file_exists(), read(), and write().

         : Dataset(base_location, db_data_location) {
                 if (filesystem::file_exists(db_data_location)) {
                         this->read(db_data_location);
                 }
                 else {
                         this->construct_dataset();
                         this->write(db_data_location);
                 }
 }

SimpleDataset::~SimpleDataset ( )

Definition at line 61 of file dataset.cxx.

61 { }

Member Function Documentation

bool SimpleDataset::add_image ( const std::shared_ptr< const Image > & image )

virtual

Adds the given image to the database, if there is an id collision, will not add the image and return false, otherwise returns true.

Implements Dataset.

Definition at line 147 of file dataset.cxx.

References id_image_map, and image().

Referenced by read().

                                                                      {
         if (id_image_map.right.find(image->id) != id_image_map.right.end()) return false;
         const std::shared_ptr<const SimpleDataset::SimpleImage> simage = std::static_pointer_cast<const SimpleDataset::SimpleImage>(image);
         id_image_map.insert(boost::bimap<std::string, uint64_t>::value_type(simage->location(), simage->id));
         return true;
 }

void SimpleDataset::construct_dataset ( )

private

Constructs the dataset an fills in the image id map.

Definition at line 70 of file dataset.cxx.

References Dataset::data_directory, id_image_map, and filesystem::list_files().

Referenced by SimpleDataset().

                                       {
         const std::vector<std::string> &image_file_paths = filesystem::list_files(data_directory + "/images/", ".jpg");
         for (size_t i = 0; i < image_file_paths.size(); i++) {
                 id_image_map.insert(boost::bimap<std::string, uint64_t>::value_type( image_file_paths[i].substr(data_directory.size(), image_file_paths[i].size() - data_directory.size()), i));
         }
 }

std::shared_ptr< Image > SimpleDataset::image ( uint64_t id ) const

virtual

Given a unique integer ID, returns an Image associated with that ID.

Implements Dataset.

Definition at line 63 of file dataset.cxx.

References id_image_map.

Referenced by add_image(), main(), and write().

                                                            {
         const std::string &image_path = id_image_map.right.at(id);
 
         std::shared_ptr<Image> current_image = std::make_shared<SimpleImage>(image_path, id);
         return current_image;
 }

uint64_t SimpleDataset::num_images ( ) const

virtual

Returns the number of images in the dataset.

Implements Dataset.

Definition at line 122 of file dataset.cxx.

References id_image_map.

Referenced by read(), and write().

                                          {
         return id_image_map.size();
 }

bool SimpleDataset::read ( const std::string & db_data_location )

virtual

Reads the specified SimpleDataset.

See write(const std::string &db_data_location) for more information about the binary format. Returns true if success, false otherwise. (checks the ifstream error bit).

Implements Dataset.

Definition at line 77 of file dataset.cxx.

References add_image(), filesystem::file_exists(), and num_images().

Referenced by SimpleDataset().

                                                           {
         if (!filesystem::file_exists(db_data_location)) return false;
         std::ifstream ifs(db_data_location, std::ios::binary);
         
         uint64_t num_images;
         ifs.read((char *)&num_images, sizeof(uint64_t));
 
         for (uint64_t i = 0; i < num_images; i++) {
                 
                 uint64_t image_id;
                 uint16_t length;
                 ifs.read((char *)&image_id, sizeof(uint64_t));
                 ifs.read((char *)&length, sizeof(uint16_t));
 
                 std::string image_location;
                 image_location.resize(length);
                 
                 ifs.read((char *)&image_location[0], sizeof(char)* length);
                 std::shared_ptr<const SimpleImage> simage = std::make_shared<const SimpleImage>(image_location, image_id);
                 this->add_image(simage);
 
         }
         return (ifs.rdstate() & std::ifstream::failbit) == 0;
 }

bool SimpleDataset::write ( const std::string & db_data_location )

virtual

Writes the SimpleDataset out to the specified file.

If the containing directory does not exist, it will be automatically created. The Dataset data is stored in a binary format with num_images() entries of the form uint64_t, uint16_t, char * corresponding to an image id, string length, and the image location string respectively. Returns true if success, fail otherwise (checks the ofstream error bit).

Implements Dataset.

Definition at line 102 of file dataset.cxx.

References filesystem::create_file_directory(), image(), SimpleDataset::SimpleImage::location(), and num_images().

Referenced by SimpleDataset().

                                                            {
         filesystem::create_file_directory(db_data_location);
         
         std::ofstream ofs(db_data_location, std::ios::binary | std::ios::trunc);
         uint64_t num_images = this->num_images();
         ofs.write((const char *)&num_images, sizeof(uint64_t));
 
         for (uint64_t i = 0; i < this->num_images(); i++) {
                 std::shared_ptr<SimpleDataset::SimpleImage> image = std::static_pointer_cast<SimpleDataset::SimpleImage>(this->image(i));
                 const std::string &image_location = image->location();
                 uint64_t image_id = image->id;
                 uint16_t length = image_location.size();
                 ofs.write((const char *)&image_id, sizeof(uint64_t));
                 ofs.write((const char *)&length, sizeof(uint16_t));
                 ofs.write((const char *)&image_location[0], sizeof(char)* length);
         }
 
         return (ofs.rdstate() & std::ofstream::failbit) == 0;
 }

Field Documentation

boost::bimap<std::string, uint64_t> SimpleDataset::id_image_map

private

Definition at line 135 of file dataset.hpp.

Referenced by add_image(), construct_dataset(), image(), and num_images().

The documentation for this class was generated from the following files:

/home/psastras/vocabtree/modules/utils/dataset.hpp
/home/psastras/vocabtree/modules/utils/dataset.cxx

Data Structures

Public Member Functions

Private Member Functions

Private Attributes

Additional Inherited Members

Detailed Description

Constructor & Destructor Documentation

Member Function Documentation

Field Documentation