--- /dev/null
+
+Last updated: 2017-04-08
+
+The FreeBSD build will build most of the tools in Ceph.
+Note that the (kernel) RBD dependant items will not work
+
+I started looking into Ceph, because the HAST solution with CARP and
+ggate did not really do what I was looking for. But I'm aiming for
+running a Ceph storage cluster on storage nodes that are running ZFS.
+In the end the cluster would be running bhyve on RBD disk that are stored in
+Ceph.
+
+The FreeBSD build will build most of the tools in Ceph.
+
+Progress from last report:
+==========================
+
+Most important change:
+ - A port is submitted: net/ceph-devel.
+
+Other improvements:
+
+ * A new ceph-devel update will be submitted in April
+
+ - Ceph-Fuse works, allowing to mount a CephFS on a FreeBSD system and do
+ some work on it.
+ - Ceph-disk prepare and activate work for FileStore on ZFS, allowing
+ easy creation of OSDs.
+ - RBD is actually buildable and can be used to manage RADOS BLOCK
+ DEVICEs.
+ - Most of the awkward dependencies on Linux-isms are deleted only
+ /bin/bash is there to stay.
+
+Getting the FreeBSD work on Ceph:
+=================================
+
+ pkg install net/ceph-devel
+
+Or:
+ cd "place to work on this"
+ git clone https://github.com/wjwithagen/ceph.git
+ cd ceph
+ git checkout wip.FreeBSD
+
+Building Ceph
+=============
+ - Go and start building
+ ./do_freebsd.sh
+
+Parts not (yet) included:
+=========================
+
+ - KRBD
+ Kernel Rados Block Devices is implemented in the Linux kernel
+ And perhaps ggated could be used as a template since it does some of
+ the same, other than just between 2 disks. And it has a userspace
+ counterpart.
+ - BlueStore.
+ FreeBSD and Linux have different AIO API, and that needs to be made
+ compatible Next to that is there discussion in FreeBSD about
+ aio_cancel not working for all devicetypes
+ - CephFS as native filesystem
+ (Ceph-fuse does work.)
+ Cython tries to access an internal field in dirent which does not
+ compile
+Build Prerequisites
+===================
+
+ Compiling and building Ceph is tested on 12-CURRENT, but I guess/expect
+ 11-RELEASE will also work. And Clang is at 3.8.0.
+ It uses the CLANG toolset that is available, 3.7 is no longer tested,
+ but was working when that was with 11-CURRENT.
+ Clang 3.4 (on 10.2-STABLE) does not have all required capabilites to
+ compile everything
+
+The following setup will get things running for FreeBSD:
+
+ This all require root privilidges.
+
+ - Install bash and link it in /bin
+ sudo pkg install bash
+ sudo ln -s /usr/local/bin/bash /bin/bash
+
+Getting the FreeBSD work on Ceph:
+=================================
+
+ - cd "place to work on this"
+ git clone https://github.com/wjwithagen/ceph.git
+ cd ceph
+ git checkout wip.FreeBSD.201702
+
+Building Ceph
+=============
+ - Go and start building
+ ./do_freebsd.sh
+
+Parts not (yet) included:
+=========================
+
+ - KRBD
+ Kernel Rados Block Devices is implemented in the Linux kernel
+ It seems that there used to be a userspace implementation first.
+ And perhaps ggated could be used as a template since it does some of
+ the same, other than just between 2 disks. And it has a userspace
+ counterpart.
+ - BlueStore.
+ FreeBSD and Linux have different AIO API, and that needs to be made
+ compatible Next to that is there discussion in FreeBSD about
+ aio_cancel not working for all devicetypes
+ - CephFS
+ Cython tries to access an internal field in dirent which does not
+ compile
+
+Tests that verify the correct working of the above are also excluded
+from the testset
+
+Tests not (yet) include:
+=======================
+
+ - None, although some test can fail if running tests in parallel and there is
+ not enough swap. Then tests will start to fail in strange ways.
+
+Task to do:
+===========
+ - Build an automated test platform that will build ceph/master on
+ FreeBSD and report the results back to the Ceph developers. This will
+ increase the maintainability of the FreeBSD side of things.
+ Developers are signalled that they are using Linux-isms that will not
+ compile/run on FreeBSD Ceph has several projects for this: Jenkins,
+ teuthology, pulpito, ...
+ But even just a while { compile } loop and report the build data on a
+ static webpage would do for starters.
+
+ - Run integration tests to see if the FreeBSD daemons will work with a
+ Linux Ceph platform.
+
+ - Compile and test the user space RBD (Rados Block Device).
+
+ - Investigate and see if an in-kernel RBD device could be developed a la
+ 'ggate'
+
+ - Investigate the keystore, which could be kernel embedded on Linux an
+ currently prevents building Cephfs and some other parts.
+
+ - Scheduler information is not used atm, because the schedulers work
+ rather different. But at a certain point in time, this would need some
+ attention:
+ in: ./src/common/Thread.cc
+
+ - Improve the FreeBSD /etc/rc.d initscripts in the Ceph stack. Both
+ for testing, but mainly for running Ceph on production machines.
+ Work on ceph-disk and ceph-deploy to make it more FreeBSD and ZFS
+ compatible.
+
+ - Build test-cluster and start running some of the teuthology integration
+ tests on these.
+ Teuthology want to build its own libvirt and that does not quite work
+ with all the packages FreeBSD already has in place. Lots of minute
+ details to figure out
+
+ - Design a vitual disk implementation that can be used with behyve and
+ attached to an RBD image.
Code Review / stor4nfv.git / blobdiff